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1.  TECHNICAL  REPORT  SUMMARY 


Thl.  document  provides  the  first  technicsl  progress  report  In  the  one-ye.r 

study  of  fsult-tolersnt  computing  which  SRI  Is  carrying  out  for  ARPA  under 

Contract  Number  N00014-72-C-1254.  The  first  hslf  ye.r  of  the  contract  Is 
covered. 


1.1.  PURPOSE  OF  THE  PROJECT 

The  general  objectives  of  our  study  are  to  evaluate  and  to  advance  the 
state  of  the  art  In  fault-tolerant  computing  systems.  The  scope  of  the 

study  Includes  theoretical  as  well  as  practical  considerations.  The  major 
task  areas  are 

(1)  to  survey  and  evaluate  existing  systems  (and  system  concepts) 
and  relevant  existing  theory; 

(2)  to  define  and  evaluate  new  directions  for  the  development  of 
computing  systems  with  high  availability  and  extensive 
fault-tolerance  capability,  with  low  cost. 
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We  are  seeking  guidelines  for  the  design  and  development  of  highly 
economical  systems  with  long  life,  near-perfect  fault  tolerance  and 
extremely  high  availability.  Understanding  of  the  tradeoffs  among  these 
goals  is  also  being  sought. 


1.2.  PROGRESS  TO  DATE 

The  study  has  progressed  on  many  fronts.  The  following  efforts  are 
considered  in  some  detail  in  Sections  3.1  to  3.11,  respectively. 


(1)  Survey  of  systems  and  system  designs  for  fault  tolerance 

(2)  Bibliography  of  relevant  literature 

(3)  Investigations  into  the  causes  of  failure 
(A)  A  hierarchical  framework  for  fault  tolerance 

(5)  Fault-tolerant  memories 

(6)  Fault-tolerant  processing 

(7)  Reliability  modelling 

(8)  Operational  implications 

(9)  Reliability  tradeoffs  among  time,  space  and  complexity 

(10)  Other  efforts 

References  are  included  to  six  project  documents  /1-6/. 


1.3.  CONCLUSIONS  TO  DATE 

We  feel  at  this  point  that  the  goal  of  obtaining  significant  fault 
tolerance  at  coats  substantially  less  than  duplication  or  triplication  of 
hardware  can  be  met  uiider  a  wide  range  of  operating  requirements. 
Particularly  In  large  systems  with  somewhat  flexible  real-time  constraints, 
the  cost  can  be  quite  low. 

Numerous  useful  techniques  exist,  some  of  which  have  been  carefully  studied 
in  recent  years.  Various  newer  techniques  which  we  are  investigating  also 
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seem  promising.  Framed-burst  coding  (in  which  bit  clusters,  or  "frames", 
are  treated  together  —  see  Section  2.5)  seems  particularly  appropriate  for 
various  advanced  technology  memory  organizations.  This  requires  only  a 
logarithmic  increase  in  the  size  of  memory  for  single-frame  burst  error 
correction  (and  a  small  Increase  in  the  overall  cost  due  to  encoding  and 
decoding).  Sparing  of  chips  on  a  frame  basis,  or  of  blocks  of  memory,  is 
also  highly  effective.  In  memory-dominated  systems,  selective  replication 
of  critical  processing  capability  may  be  used  without  greatly  affecting  the 
logarithmic  cost  increase.  This  is  greatly  facilitated  by  the  hierarchical 
framework  mentioned  in  Section  2,4,  which  appears  to  be  very  promising  in 
respects  as  well.  Interspersed  on-line  diagnostics  are  very 
important,  as  are  reliable  reconfiguration  with  sparing  and  other  concepts 
such  as  reliable  (e.g,,  distributed)  power  supplies.  In  general, 
distributed  logic-in-memory  designs  also  hold  some  promise. 


No  fundamental  gaps  in  the  state  of  the  art  have  been  uncovered  that 
prevent  the  attainment  of  high  degrees  of  fault  tolerance.  On  the  other 
hand,  up  until  now  fault  tolerance  has  generally  implied  substantial  cost. 
For  example,  hardware  cost  increases  by  factors  of  two,  three  and  four  due 
to  fault  tolerance  are  common.  (The  overall  cost  Increases  may  actually  be 
higher,  if  the  storage  required  for  fault-tolerance  software  is  considered 
along  with  the  increased  execution  time.)  The  relatively  high  cost  has 
been  a  consequence  of  some  limitations  in  the  art  that  have  hindered  the 
a^a^nmen^  economical  fault  tolerance.  Notable  among  these  is  that  it 
has  not  been  possible  to  avoid  considerable  replication  of  hardware.  As 
mentioned  above,  we  feel  that  this  can  now  be  surmounted  to  a  considerable 
degree,  except  in  systems  with  critical  real-time  requirements  on  uniformly 
correct  performance  for  all  outputs.  Nonuniformity  of  constraints  and 
requirements  can  be  used  to  great  advantage  in  system  design.  Another 
limitation  Involves  the  ability  to  achieve  practical  systems  with  long 
unmaintained  lifetimes.  This  too  now  seems  surmountable. 

In  addition,  there  are  still  some  gaps  of  understanding,  e.g.,  concerning 
space-time  tradeoffs,  the  relative  efficacy  of  replication  versus  coding  in 
arithmetic  and  logical  operations,  etc.  Furthermore,  we  are  keenly  aware 
that  the  problem  la  net  just  one  of  good  hardware  design.  The  work  of  this 
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P  is  aimed  St  providing  guidelines  for  the  design  of  good  economical 

fault-tolerant  systems.  Thus  although  good  hardware  design  is  paramount, 
oor  work  must  also  consider  Implications  of  the  system  design  in 
simplifying  the  operational  and  human  aspects,  which  play  a  critical  role 
in  keeping  .  system  highly  available.  This  Includes  considerations  that 
a  ect  the  fault  tolerance  and  the  continued  availability  of  the  operating 

system,  and  those  that  lessen  the  critical  dependence  on  skilled  operators 
and  maintenance  personnel. 

On  the  basis  of  the  work  thus  far,  we  expect  slgnlflcent  and  fayorable 
results  over  the  second  half  year  of  this  project.  We  anticipate  th.t  the 
Inal  report  will  Include  a  carefully  balanced  Integrated  approach  (or 
family  „f  approaches)  toward  achleylng  economical  systems  with  high 
availability  and  fault-tolerance.  It  Is  possible  that  as  a  result  of  this 
project  we  will  recommend  further  work  toward  the  detailed  design  and 
evaluation  of  a  specific  system.  This  system  would  presumably  be  for  a 
particular  class  of  applications,  such  as  the  network  Interface  systems  for 
a  multi-computer  network  (e.g.,  the  ARPA  network). 

The  implications  of  this  research  on  the  Department  of  Defense  Include  the 
allowing.  Systems  that  are  extensively  fault  tolerant  can  now  be 
implemented.  Such  fault  tolerance  cm  be  employed  to  yield  error-free 
computation  with  rather  long  maintenance-free  life.  The  cost  of 
fault-tolerance  Is  greatest  when  all  outputs  have  critical  reel-time 
constraints.  Otherwise  space-time  tradeoffs  permit  considerable  economy. 
Especially  for  large  computing  systems,  it  Is  possible  to  achieve  the  goals 
economically.  High  availability  Is  a  somewhat  simpler  matter,  and  can  be 
achieved  with  a  much  smaller  cost  increase. 
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2.  PROBLEMS  IN  FAULT-TOLERANT  COMPUTING 


A  system  that  is  designed  for  fault  tolerance,  high  reliability  and/or  long 
life  may  embody  various  of  the  following  functions: 

(!)  Detection  of  errors, 

(2/  Prevention  of  error  propagation, 

(3)  Location  of  faults, 

(A)  Replacement  of  faulty  units, 

(5)  Rolling  back  of  the  system  and/or  the  applications  programs. 

With  static  fault  tolerance  (e.g.,  fault  masking),  the  above  functions  are 
implemented  at  once,  in  that  a  coding  or  voting  mechanism  handles  all  five 
functions  (the  last  two  trivially).  With  most  dynamic  techniques, 
particularly  those  carrying  out  real-time  control,  the  above  functions  are 
performed  in  rapid  sequential  order  (e.g.,  within  10-100  insec  of  the 
occurrence  of  the  fault).  In  systems  with  less  critical  real-time 
requirements,  there  is  a  longer  time  period  possible  for  these  functions. 

The  five  functions  above  are  suggestive  of  the  following  problem  areas. 

(1)  Techniques  for  detecting  errors  due  to  faults  involve  the  use  of 
redundancy  (in  equipment  and/or  in  time)  to  check  the  validity  of  a  unit, 
coding  techniq1  es  efficiently  handle  the  situation  for  memory  and  siu^le 
arithmetic  units.  A  problem  exists  with  regard  to  arbitarary  control 
logic,  for  which  duplication  has  been  the  primary  vehicle  for 
fault-detection.  We  have  been  investigating  three  possibilities  with 
respect  to  control  logic.  The  first  approach  involves  realizing  most  of 
the  control  portion  of  the  processor  as  a  memory  function.  In  essence  this 
involve*  a  hierarchy  of  microprogramming  techniques  wherein  the  lowest 
levels  embody  the  highest  speed.  Memory  coding  techniques  are  used  for 
error  detection,  except  for  the  residual  arbitrary  logic  which  is  handled 
by  duplication.  The  second  approach  also  relies  upon  microprogramming,  but 
here  a  combination  of  equipment  and  time  redundancy  is  utilized  to  check  a 
computation.  With  each  microprogram  is  associated  a  checking  microprogram 
that  is  executed  following  the  main  microprogram.  A  third  approach  relies 
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upon  data-dependent  error  detection.  That  is,  we  are  looking  at  coding 
techniques  for  logic  such  that  if  a  failure  occurs  it  is  detected  with  some 
probability  p,  dependent  on  the  inputs.  Thus  if  p  is  close  to  1  on  the 
average  and  if  the  failure  is  permanent,  that  failure  will  eventually  be 

detected.  We  have  also  been  investigating  problems  on  the  use  of  feedback 
in  detection. 

(2)  The  prevention  of  error  propagation  takes  several  forms.  One  involves 
the  use  of  replication  and  voting,  another  the  use  of  error-correcting 
coding.  Another  approach  is  to  resort  to  delay,  by  refusing  to  give  any 
output  at  all  until  a  guaranteed  correct  output  can  be  obtained. 

(3)  The  location  of  a  failure  requires  that  some  form  of  diagnostic 
procedure  be  carried  out  subsequent  to  the  failure  detection.  With  regard 
to  memory  failures  we  have  been  looking  at  frame-error-locating  and 

correcting  codes  and  at  more  conventional  diagnostic  procedures. 

Similarly  we  have  briefly  investigated  byte-locating  arithmetic  codes. 

W  In  approaching  the  problem  of  faulty  unit  replacement,  the  first  step 
is  to  identify  a  suitable  partitioning  of  the  system.  The  partitioning  can 
be  accomplished  at  the  level  of  an  entire  processor  (albeit  a  small  one), 
or  at  the  level  of  memory  blocks,  or  of  frames  within  a  word.  For  this 
case  the  replacement  is  quite  easy,  requiring  only  a  modification  in  the 
address.  However,  reliable  switching  is  very  important.  For  the  computer 
utility  application,  a  large  processor  may  be  too  expensive  to  represent  a 

viable  discardable  unit,  so  that  a  greater  number  of  smaller  processors  may 
be  desirable. 

(5)  With  regard  to  real-time  systems,  most  proposed  fault-tolerant  systems 
incorporate  single-instruction  rollback.  General-purpose  utilities  usually 
let  the  user  worry  about  his  own  rollback.  For  real-time  applications, 
automatic  program  restart  and  system  restart  are  essential.  Fortunately, 
the  nature  of  the  environment  usually  permits  them  to  be  implemented 
easily.  For  the  utility  environment  it  also  seems  that  user-invisible 
rollback  can  be  effectively  achieved.  For  example,  a  compiler  might  select 
rollback  points  where  the  pertinent  status  information  can  be  automatically 
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checkpointed.  However,  there  is  a  severe  problem  in  recovering  from  major 
catastrophes  such  as  power  failures  and  certain  critical  hardware 
malfunctions.  System  rollback  is  clearly  more  difficult  in  an  unknown 
environment. 

(6)  Analysis  of  system  design  is  another  important  problem  area. 
Evaluation,  e.g.,  via  design  verification,  modelling,  simulation,  is 
particularly  important  in  terms  of  reliability,  fault  tolerance  and 
availability. 

(7)  The  development  of  systems  with  critical  constraints  can  be  difficult. 
There  is  a  need  for  structured  system  design  to  facilitate  the  development 
process.  The  hierarchical  framework  of  Section  3.4  aids  greatly  in  this 
respect. 


3.  SUMMARY  OF  PROGRESS 


The  following  paragraphs  describe  progress  on  specific  work  being  conducted 
under  this  project.  Essentially  all  tasks  are  directly  related  to  the  goal 
of  obtaining  as  much  fault  tolerance  as  possible  for  as  little  cost  as 
possible,  commensurate  with  the  nature  of  the  system  requirements. 


3.1.  SURVEY  OF  SYSTEMS  AND  SYSTEM  DESIGNS  FOR  FAULT  TOLERANCE 

A  census  has  been  made  of  fault-tolerant  systems  and  system  designs.  A 
first  version  is  given  here  as  Appendix  I,  and  includes  a  very  superficial 
summary  of  each  system.  To  be  able  to  represent  systems  in  a  more-or-less 
canonical  way,  we  have  designed  a  questionnaire  (included  in  Appendix  II), 
which  we  have  sent  to  architects  of  most  of  these  systems.  The  replies 
received  thus  far  are  given  in  Appendix  II.  Most  of  them  contain 
significant  detail,  and  permit  ready  comparison  of  the  various  goals, 
motivations,  principles,  techniques  and  achievements.  The  design  of  the 
questionnaire  itself  has  exposed  many  dimensions  of  meaningful  comparison 
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among  systems  and  reflects  many  different  design  approaches  based  on  widely 
varying  goals  and  constraints.  (An  earlier  version  of  the  survey  was 
distributed  at  the  Second  International  Symposium  on  Fault-Tolerant 
Computing,  held  in  Boston,  June  19-21,  1972.  It  formed  the  basis  for  a 
well-received  panel  discussion  entitled  "Approaches  to  the  Architecture  of 
Fault-Tolerant  Computing",  chaired  by  Jack  Goldberg  and  including  John 
Wensley  as  a  panelist.) 

In  order  to  classify  the  numerous  fault-tolerance  architectures,  we  have 
selected  three  categories  of  systems,  corresponding  to  three  roughly 
disjoint  applications  areas: 

(1)  General-purpose  computing  utilities, 

(2)  Ground-based  special-purpose  systems 

(3)  Aerospace  systems 

The  processor  power  and  total  system  cost  are  more  or  less  decreasing  from 
(1)  to  (3),  as  is  the  size  of  memory  required.  The  degree  of  preplanning 
possible  for  the  computations  generally  increases  in  this  order.  The 
reliability  and  availability  requirements  usually  increase  in  criticality 
from  (1)  to  (3),  (RELIABILITY  is  the  probability  that  the  system  will 
perform  satisfactorily  for  at  least  a  given  period  of  time  when  used  under 
stated  conditions.  AVAILABILITY  is  the  probability  that  the  system  is 
operating  satisfactorily  at  any  point  in  time  when  used  under  stated 
conditions,  where  the  total  time  considered  includes  operating  time,  active 
repair  time,  administrative  time,  and  logistic  time, . .RELIABILITY 
ENGINEERING,  ed.  W.  H.  von  Alven,  Prentice  Hall,  1964,  pp  14-15.) 

Kcst  of  the  prior  research  efforts  have  been  devoted  to  category  (3).  We 
feel  that  the  problems  in  this  category  are  basically  solved.  Replication, 
multiprocessing,  coding,  and  sparing  are  commonly  found  techniques.  While 
the  relative  cost  of  fault-tolerance  is  high  in  many  of  these  systems,  this 
is  not  always  necessery.  We  feel  that  many  of  these  systems  may  be 
over-engineered. 

The  situation  is  less  well  developed  for  the  other  two  categories,  which 
still  seem  relatively  primitive  (cf.  the  Census  of  Appendix  I). 
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Intuitively,  the  relative  cot t  of  fault  tolerance  could  be  substantially 
less  than  in  category  (3),  because  of  the  looser  constraints,  and  because 
of  the  possibilities  of  advantageously  using  averaging  effects  in  bigger 
systems.  This  report  tends  to  justify  this  statement. 


3.2.  BIBLIOGRAPHY  OF  RELEVANT  LITERATURE 

In  March  1968  R.  A.  Short  published  a  rather  comprehensive  bibliography  /7 / 
(containing  347  references)  which  resulted  from  an  SRI  study  for  NASA.  He 
is  now  helping  us  augment  that  bibliography  with  about  500  additional 
references.  A  system  of  descriptors  and  cross-indices  is  expected  to  be 
used  which  will  greatly  enhance  the  usefulness  of  the  bibliography. 
References  to  systems  mentioned  in  this  report  are  found  in  Appendices  I 
and  II. 


3.3.  INVESTIGATIONS  INTO  THE  CAUSES  OF  FAILURES 

Early  investigations  have  led  to  various  (sometimes  obvious)  conclusions 
regarding  the  significant  sources  of  system  failures. 


*  Magnetic  core  memories  operated  with  low  access  times  (one  microsecond  or 
less)  are  major  sources  of  system  failures.  Primary  memory 

still  dominates  most  systems  (especially  large  ones)  with  respect  to  cost, 
size,  and  sources  of  unreliability. 

*  Peripherals  are  still  a  problem,  although  good  system  design  should  be 
able  to  prevent  peripheral  failures  from  "crashing"  the  system.  (Several 
well-known  systems  are  quite  sensitive  in  this  area,  due  to  poor  system 
design.  For  example,  a  system  should  be  able  to  survive  errors  in  reading 
most  files,  a  capability  which  is  facilitated  by  hierarchical  design.) 

*  In  several  technologies  transient  or  intermittent  faults  are  more 
significant  (and  more  common)  than  permanent  faults  (e.g.,  "stuck-at") , 
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although  the  latter  are  more  commonly  considered  in  the  literature.  These 
arise  in  many  ways,  e.g.,  timing  errors,  data-dependent  faults,  and 
marginal  design.  Correlated  faults  are  also  problematic  in  practice,  both 
operationally  and  in  terms  of  analysis.  For  example  in  an  LSI 
implementation,  a  single  chip  failure  may  result  in  multiple  chip-output 
errors.  Physical  couplings  are  still  a  major  source  of  difficulty,  both 
in  bonding  and  in  pin  connections. 

*  Problems  in  operations  and  in  the  inner-core  of  the  operating  system  are 
a  major  source  of  trouble  in  large  computer  utilities.  Even  if  the 
hardware  were  faultless,  there  are  still  enormous  problems  in  keeping  such 
a  system  cperational  with  high  availability.  These  problems  are 
distributed  among  weak  or  inadequate  software  designs  and  questionable 
operating  practices,  as  well  as  an  occasional  hardware  design  error.  Two 
notable  recent  cases  involve  a  ten-hour  outage  of  an  ESS  No.  1  installation 
and  a  29-minute  outage  of  the  NY  Stock  Exchange  MDS-1.  Both  systems  are 
designed  for  fault  tolerance  and  high  availability,  but  both  experienced 
major  outages  attributable  to  human  frailty  (maintainers  and  system 
programmers,  respectively)  that  aggravated  troubles  due  to  hardware 
difficulties.  (Good  system  design  can  help  circumvent  these  problems.) 


3. A.  A  HIERARCHICAL  FRAMEWORK  FOR  FAULT  TOLERANCE  ' 

Peter  Neumann  has  formulated  a  hierarchical  framework  for  fault-tolerant 
computing  systems,  relevant  to  both  hardware  and  software.  It  i*  described 
in  a  paper  to  be  presented  at  COMPCON  72  in  September  /2/,  included  here  as 
Appendix  III.  This  framework  facilitates  the  dynamic  alteration  of 
fault-tolerance  techniques,  as  best  suited  to  the  current  computing  needs. 
This  holds  great  promise  for  the  attainment  of  economical  fault  tolerance, 
especially  if  real-time  constraints  are  not  uniformly  critical.  It  is 
immediately  applicable  to  large  computing  systems,  and  is  useful  to  small 
systems  as  well.  The  hierarchical  framework  greatly  facilitates  the 
control  over  the  exchange  of  redundant  equipment  for  occasional  slight 
increases  in  time,  as  well  as  enhancing  the  development  and  operation  of 
the  system. 
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3.5.  FAULT- TOLERANT  MEMORIES 


Several  investigations  «re  under  way  Involving  fault  tolerance  in  memories. 
Memory  organizations  suitable  for  advanced  technologies  are  being 
investigated.  Multiple-bit-per-chip  ("f rame-per-chip")  organizations  seem 
very  powerful,  and  make  framed-burst  coding  (also  called  phased-burst 
coding)  highly  advantageous.  Peter  Neumann  has  examined  such  an 
organization,  and  shown  that  such  coding  can  be  highly  effective  and 
economical  /3/.  Karl  Levitt  has  incorporated  these  concepts  into  the  BUCS 
(Bus  Checker  System)  design  /8/,  which  appears  to  offer  good  reliability  at 
low  cost.  Although  designed  to  take  advantage  of  an  aircraft  environment, 
its  balance  of  design  seems  to  have  significant  implications  for  our  goals. 
[Such  coding  techniques  are  also  found  in  MDC  (see  Appendix  I  and  II).] 

John  Wensley  has  been  examining  the  problem  of  reliably  reconfiguring  at 
the  chip  level  in  a  large  memory  requiring  many  chips  (somewhat  akin  to  the 
problem  of  making  page  relocation  mechanisms  reliable).  One  rather 
promising  solution  involves  distributed  control.  Also  applicable  is  some 
earlier  work  in  our  Computer  Science  Group  /9 /  on  reliable  switching, 
useful  for  example  in  the  switching  of  spare  frames.  In  a  related  effort, 
Jack  Goldberg  has  been  studying  distributed  processor  designs  for  fault 
tolerance  /A/. 


3.6.  FAULT-TOLERANT  PROCESSING 


We  have  been  investigating  fault  tolerance  for  logic  operations  and  for 
arithmetic  operations.  In  systems  which  are  memory  dominated,  replication 
of  processing  units  may  be  reasonably  economical.  In  other  situations 
coding  may  be  desirable.  Peter  Neumann  has  shown  / 5/  that  single  faults 
can  be  detected  in  logic  operations  by  performing  all  arithmetic  logic 
operations  in  an  arithmetic  unit  of  suitable  design,  without  additional 
(redundant)  circuitry.  Monteiro  and  Rao  / 10/  have  recently  investigated 
arithmetic- logic  units  with  greater  fault  coverage. 
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3.7.  RELIABILITY  MODELLING 


John  Wensley  has  been  investigating  existing  work  on  reliability  modelling 
and  its  relevance  to  fault-tolerant  computer  design.  Reliability  modelling 
for  fault-tolerant  computing  has  been  the  subject  of  several  studies 
/11-15/.  Models  have  been  proposed  and  analysed,  but  in  general  they  do 
not  answer  certain  important  questions  that  arise  in  consideration  of  many 
fault-tolerant  computers.  We  note  some  of  these  deficiencies  here. 

Existing  models  are  more  concerned  with  "survivability"  than  with 
reliability.  In  general  the  assumption  is  made  that  when  a  fault  occurs  it 
totally  disables  the  unit  in  which  it  occurs,  and  that  that  unit  must  then 
be  removed  from  future  computational  significance  (possibly  being  replaced 
by  a  spare),  or  its  significance  removed  by  such  techniques  as  voting  in  a 
TMR  system.  This  does  not  allow  accurate  modelling  of  transient  faults,  in 
which  ?.  unit  may  survive,  but  some  data  may  have  been  corrupted.  In 
addicion  such  models  do  not  handle  repetitive  transients  (intermittents) 
and  the  loss  of  effective  computer  power  due  to  the  CPU  activity  involved 
in  error  detection,  correction,  masking,  etc. 

^  further  deficiency  of  such  models  is  the  common  assumption  that  any 
correction  of  a  fault  (by  sparing,  reconfiguration,  masking,  etc.)  can  be 
modelled  by  a  single  parameter,  representing  the  probability  of  correct 
recovery  from  a  fault.  This  does  not  allow  the  modeller  to  distinguish  for 
example  between  faults  that  can  be  removed  by  voting  and  those  whose 
erroneous  effects  cannot  be  so  removed  (e.g. ,  a  mistyped  line). 

The  last  deficiency  to  be  noted  is  that  the  models  do  not  take  into  account 
the  fact  that  faults  have  different  effects  depending  on  the  state  of  the 
Bystem  at  the  time  of  the  fault.  In  some  computer  systems  a  dynamic 
trade-off  is  possible  between  reliability  and  computing  power  (q^>.  SIFT, 
ARMMS) .  Faults  that  occur  in  the  high  reliability  mode  are  less  serious 
than  those  that  occur  in  the  high  computing-power  mode. 

The  existing  modela  have  further  drawbacks.  However  the  above  discussion 
clearly  shows  a  need  for  research  aimed  at  developing  modelling  techniques 
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that  can  handle  some  of  the  specialized  problems  arising  In  the  analysis  of 
fault-tolerant  computers. 


The  consequence  of  the  above  Is  that  the  reliabilty  estimates  may  be  very 
pessimistic,  which  could  result  in  very  costly  over-enginaered  designs.  It 
is  clear  that  improved  methods  of  assessing  reliability  are  required. 

3.8.  OPERATIONAL  IMPLICATIONS 

Various  implications  of  system  design  on  system  reliability  have  been 
considered,  in  response  both  to  problems  arising  in  existing  systems  and  to 
problems  introduced  by  newer  designs.  As  an  example,  John  Wensley  has 
examined  the  existing  time-shared  computer  systems  at  SRI,  (both 
ThNEX  systems),  to  ascertain  the  posslbilty  of  utilizing  an  automatic 
checkpoint  scheme  is  such  environments,  A  conceptually  simple  scheme 
consists  of  simultaneously  recording  on  magnetic  tape  all  data  that  is 
normally  placed  on  drums  or  disks.  The  saving  of  such  data  would  provide  a 
continuous  history  of  the  state  of  the  users  files  so  that  unavailability 
of  user  files  due  the.  loss  of  a  disk  or  drum  could  be  circumvented. 

Initial  studies  show  that  a  magnetic  tape  system  of  quite  modest 
performance  could  handle  the  total  data  currently  written  onto  drums  and 
disks.  (In  one  test  the  average  over  31/2  hours  was  a  bandwidth  of  15 
Kch/sec,  which  is  well  within  the  state  of  the  art  of  magnetic  tape 
systems.  With  an  assumption  of  50  per  cent  efficiency  of  tape  utilization, 
a  single  2400  ft.  tape  could  store  the  total  traffic  of  24  minutes  of 
computer  activity.)  Here  good  design  in  input-output  hardware  can  be  of 
great  help,  permitting  simultaneous  writes. 


In  another  direction,  Multics  incremental  backup  experience  shows  half-hour 
delayed  backup  is  very  helpful  to  users  in  the  event  of  system  crash. 
However,  recovery  after  damage  to  certain  system  files  may  be  a  very 
lengthy  procedure.  Combatting  this  problem  requires  very  careful  system 
design  in  a  distributed  system  such  as  Multics.  Another  problem  in  Multics 
is  introduced  by  operator  errors  in  performing  manual  reconfiguration. 
Significant  care  in  hardware  design  can  aid  considerably  in  such  problems. 
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3.9.  RELIABILITY  TRADEOFFS  AMONG  TIME,  SPACE.  AND  COMPLEXITY 

DAFid  Huffman  haa  bean  inve.tig.ting  the  Implication  of  information 

kn“  0"rehllabllU!'  Md  *«**  ^laranca.  Coding  tbaoty  show.  that 

.fFici.ncy0„ftreeiiUbTM  ‘  C°m"UnlCMl0"  »"»“  —  Steady  improve  tbe 
reliab,  communication.  Similar  fault,  are  being  a ought  ,or 

t  ltTmPUta:l0n'  *  "all8tlC  "0del  1S  h*1"*  developed,  ao  that 

results  may  have  some  practical  significance. 

trld'T  °f  lnVMtl8Mlon  fe  effect,  of  time-ap.ce 

12°“’ r  rellabUlty*  SU8ht  •*  critical  real-time 

hardware  «,‘i““  for"’^  1^1' the  CMt  of 
problem  better  (It  n,  *  *”  be8inning  to  underatand  thia 

-It  ia  M  “I  ^  — 

nis  formalisms  can  be  of  help. 

Also  related  is  our  desire  t-n  .k.  . 

tradeoffs.  The  results  of  c  3  "  re8UltS  °n  comPlexlty-reliability 
some  highly  unrealistic  assu^ption^  ^hlv!  30,1  ^  EUaS  h®86*1  °" 
-ingful  results  with  more  realist,  Is^tZ.  ^ 


3.10.  OTHER  EFFORTS 


We  are  also  devoting  some  efforr 

transient  recovery. problem,  in  which  a  ^  ^  ^  maSSlVe" 

(o.E..  a  power  anrge)  bM  left  all  8°"rCe 

of  Appendix  III).  Another  problCT  ls  „  be  “’'“em  anapect  (aee  the  end 

how  to  dlagnoae  normally  unuaed  but  critlc.T  Unlt"  Pr0‘,lem'  ll‘-  °f 

(hardware  and/or  software)  as  f  Portions  of  the  system 

‘-If.  The  general  rol”  If  “T*  'h'  —very  proceaa 

Elapa-  /6/  hea  earned  an  extend™  of  th°1S°  h*1"8  C0"Sldered-  B»™ rd 
Preparata,  Metre  and  chlen  This  *  "°<ial  f°r  self"“lae"oai8  of 

reaulta.  '  IhlS  “te"sl°"  h°3«a  acme  promiae  for  uaeful 
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APPENDIX  J. 

CENSUS  OF  FAULT-TOLERANT  COMPUTING  SYSTEMS 
(SRI,  first  version,  August,  1972) 


Following  is  a  list  of  systems  and  system  designs  providing  significant 

and/°r  avallablllty-  Those  systems  indicated  by  ’*($)"  are 
tZitl  f  lnAgrea^ar  daPail  our  Survey  of  Fault  Tolerant  Computing 

aie  described  r  J  >f  ’f16™  references  fire  included.  Several  systems 

S  Miller  n  r  feferred  to  here  as  the  "Intermetrics  Report"  (J. 

*  '  *  Sickly,  A.  L.  Kosmala  and  J.  A.  Saponaro.  Multiprocessor 

r*mhw*r  CM  y’  inal  Report*  Contract  NAS  9-9763,  Intermetrics,  Inc., 
here  Mass»  MaJ;ch*  1970>*  °ther  systems  are  given  terse  references 

ere,  where  available.  Aobreviations :  P  -  Processor,  M  -  Memory,  SEC  - 
single  error  correction,  (D)ED  -  (double)  error  detection. 


A.  GENERAL-PURPOSE  COMPUTING  UTILITIES,  generally  good  availability,  human 
usets,  modest  reliability,  maintenance  permitted 

1.  Multics  MIT  and  Honeywell,  Cambridge,  Mass  (F.  J.  Corbato);  ARPA- 
funded  development.  See  E.  I.  Organick,  A  Guide  to  Multics,  MIT  Press. 

General-purpose  computing  utility  (time-sharing,  batch),  with  high 
availability  and  very  high  file  integrity.  Four^ysteL  currently8 
operating,  J 

of2p!  (G^-»oneyweH  645s) s  multiprocessed,  manual  on-line  reconfiguration 
of  Ps  and  Ms,  extensive  fault  isolation  via  the  ring  mechanism  for 

fileebaik,nnanv  ^ fil*  3y!tP"'  aCC6SS  Control»  half-hour  lag  incremental 
file  backup,  variable-depth  .system  rollback,  redundancy  in  the  file 

St5UKtyre\  (Sln8le  ED*  minimal  error  checking  in  most  systems, 
not  mentioned  below.)  7  » 


2($).  PRIME  (nee  MCS),  University  of  California  at  Berkeley  (H.  Basking 

*  ReTiable,  secure,  modest  computer  utility,  high  availability.  In 
development. 

*  5  P  practlcal  f°r  3  P  to  8  P),  with  highly  restricted  possible 

connectivity  of  M  and  disk,  strict  isolation  with  no  memory  sharing  or 
multiprogramming,  spontaneous"  reconfiguration  via  a  reliable 
self-checking  switch.  About  10%  overhead  for  fault-tolerance. 


3.  Carnegie-Mellon  University;  NSF. 

*  Research  system  development  with  applications  to  ARPA  speech 
understanding  project;  in  design 

*  16  P  x  16  M  (PDP  11s),  with  reliable  crosspoint  switch 


4.  University  of  Newcas tle-on- Tyne ,  England;  Scientific  Research  Council. 

*  General  computing;  in  design 

*  PDP  11s 
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5.  Various  commercial  time  sharing  services  gain  availability  (out  not 
necessarily  reliability)  by  having  cross-switchable  Ps ,  Ms  and  secondary 
memory,  J 


B.  GROUND-BASED  SPECIAL  PURPOSE  SYSTEMS,  controlling  the  environment  (or 
controlled  by  it),  generally  higher  reliability  and  availability,  often 

pofsible^31  C°nstralnts  than  in  A  aboVe.  bumfln  maintenance  usually 

6($).  ESS  (Electronic  Switching  Systems),  Bell  Labs,  Naperville,  lllinos. 

Telephone  switching  system;  long-term  continuous  availability,  with 
occasional  errors  tolerable  to  customers  (?).  Over  200  Number  1  ESS  in 
operation,  many  more  Number  2  ESS,  TSPS, 

*  2  Pcf1  functlonal*  1  standby  checking  and  diagnosis),  automatic 
reconfiguration.  Separate  nonalterable  program  store  with  SEC.  50%  of  all 
programs  are  diagnostics.  Millions  of  hours  of  experience  have  aided  in 
improving  hardware  and  software  reliability.  People  problems  still  very 
difficult  (operations,  maintenance). 


7.  FAA  (Federal  Aviation  Admin.),  IBM.  See  IBM  Sys  J. ,  vol  6,  no  2,  1967. 

Air  traffic  control,  long-term  continuous  availability.  Untolerated 
nontransient  errors  can  be  disastrous.  20  systems  at  ATC  centers  covering 
the  continental  United  States. 

*  Up  to  A  P  (IBM  9020),  up  to  12  M.  Propram-controlled  error  analysis  and 
reconfiguration,  gracefully  deconfigurable.  5-second  battery  backup  power 
supply.  Relies  heavily  on  good  and  highly-available  field  engineers. 


8.  MDS-2  (Market  Data  System),  New  York  Stock  Exchange 

*  Stock  trading  ticker  control.  Near-continuous  availability,  no 
transaction  losses  permitted.  Operational  August  1972.  Precursor  MDS-1 
operational  for  7  years. 

*  3  P  (360/50),  2  multiprocessing  with  shared  M  &  LCS  (but  1  P  basically 
monitoring),  3rd  P  normally  spare  (while  running  background  lobs), 
extensive  program  checking. 


9($),  COMEX,  Pacific  Coast  Stock  Exchange 

*  Stock  trading  control;  near-continuous  availability,  no  transaction 
osses  permitted,  small  real-time  lag  permitted.  Operational  since  1969; 

2  complete  systems  (each  has  360/50  plus  2  PDP  8s),  one  in  San  Francisco, 
one  in  Los  Angeles,  capable  of  running  separately  or  cross-switched 
(interconfipurable) . 


10.  NASDAQ,  National  Association  of  Securites  Dealers  Automated  Quotations 
See  Datamation,  March  1972,  pp.  42-45, 

*  interactive  system  to  facilitate  trading  of  OTC  securities;  high 
availability;  operational  since  end  of  1971. 

*  2,P  (110®s)»  multiprocessing  under  EXEC  8,  capable  of  running  simplex. 
Dual  records  in  file  structure,  automatic  recovery  techniques. 
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11.  CLC,  Bell  Labe.  Uhlppany  NJj  ABMDA  (Safeguard) 

reeJlred  lfr11?  defenSei  co"tl"“>“  availability  when  (and  If) 
required.  In  development  since  mid-60s. 

sae  r  vssr-  • 

only!®  ^  8  r  d6eP  8paCe  ProbeS-  Availability  not  critical.  Design 

v^r-^aid^lilar0”1 

Revlew^^Feb *  1 ' 70dD^p^^l-x°n8  Ub'  H"l0U-  E"ela"d-  S"  E1"»1<=»1 
*  Real-time  control 

Vi  of  M^access ^svi tches • 't rloll* ' 1 1""  *?  I*°!  du',llc“tl°"  »f  puneh/reader 
522  of  hardware  due  to  flult  folerlnce  “ntt01  and  °f  funCtl°n  “"U- 

14.  Foxboro  88.  Foxboro  Corp.  Procesa  eontrol  using  2  P  (PnP  8s) 

~,”Jsc*  SSS'eSSiS  S2  tiltra‘hl5h  rllabIllty  aad  ««*‘i*t» 

Install  Y*  r°SSlhla-  “  laa,t  the  ef forts^have^resulte^ln 

eff°rtS  "«1’ 

S  rjrr  Lab  (A ■ L- 

Apollo  guidance.  Used  in  Apollo  program. 

1  p  (I  as  standby),  M  duplicated. 


16($).  JPL-STAR,  JPL,  Pasadena  Cal  (A.  Avizienis);  NASA 

rriy  sysx:;g:  ^  ln 

triplicated  monitoring  and  control  (TARP  -  test  and  wnair  *\ 
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i"fr»arK'  “C"  MIT  Drai’er  Lab-  Ca"b'‘<M,  Mass  (A.L.  HopKlns , 

£.a„“.“dP"-  “Rh  rallabl“*  *■» 

units,  «pll€StionS»ithinneIch“JIroce!si’nEUinnrandSuUhia,I°n8  pr°“aaln8 
coding) ,  Two  concepts:  P  18  iC  nd  wlthin  memories  (without 

memories  end  buses”<!lth™JIres"1',laXad  acratchpaii  "emeries,  triplexed 

-L"^“:L^rslnf-er>tcbpad  ”ita- 

*bGeneral-purpo8eCdeslgnUforasoeeiaiaSaUdt '  **•  C1°“d-  »« 

aerospace.  Prototype  now  working  !’urpoaa  applications,  including 

representations* (withal101**18  '"s'  (7-<,)  “  DED  °”  decimal 
reconfiguration.  About  66:TMdl!ndantlnatl°nS>  ’  8parln8>  “i'roprogrammable 


NASA-Huntsville*1**  dlfltal  C0"P“tar)'  IB«  8»rktow„  Hts,  NY, ; 

*  m^p^multiprocessing 

0nende!rL\r1a„r:ol£aL1r;„sfsai1b1:rfeMi“^i  tbird> in  4  8  *a“i°!"“ 

srrors  handled  in  M,  eVt«°nsP“eSl.be!i-ch!X?SnOStl“’  b-adJa“Pt  ”‘“ltlpla 

—  « 

reliablliVy.  and  conerol,  space  shuttle  use;  long-life 

5-  apbp  -d  suh„  , 

switchable  via  "ripnler”  b.,r«?g‘  SEC,in  M  Plus  3  spare  bits  reliably 
control.  d„plic.«r^„rfigbUr.r:itoTcr^rt^;:Ctl0n  “  “  M>  “lpUaatad 
The  Ultrasystems  entry  is  similar  to  the  JPL  STAR. 

nIsa-uS  S°ft“are  ln,pleMntad  fa““  Colerance,  SRI  (John  Wensley) . 

during  flight;  some ( tasks rmo re  crltical^han’otheri1'’’  0f  correct  results 
degradation  of  less  critical  tasks.  ’  Permitting  slight 

*  Multiprocessing  with  variable  software  repliraMnn  a  a 
application  program  (software  reconfigurable) ! 
software  techniques  avoids  need  for  sLh.i  I  1  tolerance  via 
existing  designs.  Connectivity  is  re^tric^ed"^"6’  P!|J^it8  U8e  °f 
can  read  all  Ms,  preventing  fault  nronaa  «  d‘  PrCan  modify  0nly  its  own  M, 

fault-tolerance  procedures  ^asthe  aonH?\<  ‘  Executlve  »«  the  same 
redundant  °ceaures  as  the  application  programs.  About  75% 
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22($).  ARMMS,  Hughes,  Fullerton  CA  (W.  L.  Martin);  NASA-Marshall  (MSFC) 

*  Spacebome  control;  long-life  reliability 

n  P,  dynamically  reconfigurable,  e.g.,  as  independent-process 
multiprocessing  or  as  replication  with  sparing.  20%-80%  redundant 
(variable) 

23($) ,  Intermetrics  multiprocessor,  Cambridge  Mass  (J.  S.  Miller) 
outgrowth  of  EXAM;  NASA-ERC  (Houston)  ' 

*  Manned  orbiting  space  station 

*  m  P  (1  to  8,  nominally  3),  each  P  duplicated,  coding  in  M  (ED),  buffered 
instruction  retry,  save  within  interrupted  instruction. 


24($).  Autonetics  (N.  Am.  Rockwell,  Anaheim,  L.  J.  Koczela) ;  NA5A-MSC 

*  Space  shuttle;  long-life  reliability 

*  4-level  redundancy,  FO-FO-FS  (cf.  MDC)  requires  80%  redundancy,  less  for 
lower  fault  tolerance. 


25.  BUGS  (bus  checker  system),  SRI  (Karl  Levitt),  NASA-Langley . 

See  SRI  Final  Report,  NASl-10920,  1972  (Reference  8  of  this  Report). 

*  Aircraft  control,  as  in  SIFT 

*  5_1°i^0Cf1^  P  &  M  units,  each  duplicated  internally,  frame  coding  in 
central  M,  bus  checker  coordinates  restart  mechanism,  periodic  diagnoses  of 
M  and  of  unflexed  processor  functions.  About  33%  redundancy. 


VopS.  JPL  (Gilley)‘  See  IEEE  Trans.  Astr-Aero,  Sept.  1970. 

*  Thermo-electric  outerplanet  space  travel 

*  Related  to  JPL-STAR. 


27.  MFC,  Hamilton-Standard;  NASA-ERC.  See  Intermetrics  Report. 

*  Modular  flight  computer 

*  3  P,  3  M,  cross-configurable,  TMR  or  3  P  multiprocessor 


28.  ALPHA,  CDC.  See  Intermetrics  Report. 

29.  AADC,  Honeywell;  NASA,  AADC  Naval  Air  Systems  Command.  See 
Intermetric8  Report. 


30.  IRAD,  Litton,  See  Intermetrics  Report. 

31.  SDC-Burroughs ;  USAF-Wright-Patterson,  Multiprocessor 

32.  S-3,  Univac 


33.  COSMOS,  RCA  (cf.  SUMC) 
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APPENDIX  II 

SURVEY  OF  FAULT- TOLERANT  COMPUTING  SYSTEMS  (revised  Aug  1972) 

Jack  Goldberg,  Peter  G.  Neumann  and  John  H.  Wens  ley 
Computer  Science  Group,  SRI,  Menlo  Park,  CA,  94025 

This  appendix  presents  replies  to  a  questionaire  sent  to  architects  of 
various  fault-tolerant  computing  systems.  It  is  hoped  that  the  questionaire 
wiil  itself  be  useful  as  a  descriptive  form  and  that  the  replies  will  aid 
in  understanding  and  comparing  the  systems  included  here.  To  this  end  the 
questionaire  has  been  designed  to  permit  a  concise  description  of  each 
system,  its  goals,  its  motivations,  its  principles,  its  structure,  its 
techniques,  and  its  achievements  to  date. 

The  first  issue  of  this  document  was  distributed  informally  to  conference 
participants  at  the  Second  Symposium  on  Fault  Tolerant  Computing,  Boston, 
June  19-21,  1972.  It  was  intended  to  support  the  panel  discussion 
Approaches  to  the  Architecture  of  Fault-Tolerant  Computing",  chaired  by 
Jack  Goldberg.  The  replies  given  here  are  included  essentially  in  their 
entirety.  Significant  efforts  not  represented  here  Include  several 
existing  systems  (such  as  IBM's  FAA  system,  Bell  Lab's  CLC  and  various 
query  systems)  as  well  as  numerous  design  and  development  efforts  (e.g. 
government  systems,  and  systems  under  development  at  Carnegle-Mellon 
University  and  the  University  of  Newcastle-on-Tyne) . 

The  contents  of  this  appendix  are  as  follows. 

Questionnaire  njlo_  j  ■, 


Replies  of  the  panelists: 

A.  Avizienis,  JPL  and  UCLA  24-25 

W.  C.  Carter,  IBM  26-27 

A.  L.  Hopkins,  Jr.,  MIT  Draper  Lab  28-29 

W.  L.  Martin,  Hughes  Aircraft  30-31 

J.  H,  Wensley,  SRI  32-33 

Other  replies: 

B.  R.  Borgerson,  U.  C,  Berkeley  34-36 

J.  L.  Delamare,  EMD,  France  36-37 

L.  J,  Koczela,  North-Araerican  Rockwell  38 

J.  S.  Miller,  Intermetrics  39 

D.  C,  Wallace  (SRI)  for  PCSE  40-41 

W.  Ulrich,  Bell  Labs,  Naperville,  Illinois  42 

Capt.  L.  A,  Fry,  SAMSO,  Los  Angeles,  Ca  42 


The  preparation  of  this  document  and  the  questionaire  was  supported  by  the 
Defense  Advanced  Research  Projects  Agency  of  the  Department  of  Defense,  and 
was  monitored  by  ONR  under  Contract  Number  N00014-72-C-0254.  The  views  are 
clearly  those  of  the  named  contributors  and  do  not  necessarily  represent 
any  official  policies  of  ARPA  or  the  U.  S.  Government. 


22 


StIRVXT  or  FAULT- TOLERANT  COMPUTING  SYSTEMS— QUESTIONNAIRE 
’tck  Goldberg  and  Peter  G.  Neumann,  SRI,  Menlo  Park  CA 


of  the  system  (and/or 


1.  IDENTIFICATION  of  tha  ayata. 

1.1.  NAME:  What  la  tha  ralavant 
project)? 

1.2.  RESPONSIBILITY!  Wliat  la  tha  responsible  organization? 

1.3.  SUPPORT!  What  ara  tha  aourcaa  of  aupport? 

1.4.  PARTICIPANTS!  Who  (and  what  organizations,  if 
ralavant)  ara  tha  principal  participants? 

1.5.  START:  What  waa  the  data  of  conception? 

or  11  “P,ct*d  to  >>•.  tha 

ccatplation  data?  (Specify  prototype  acceptance  data,  or 
dcalgn  completion  data  If  dealgn  only.) 

1.7.  BIBLIOGRAPHY!  What  ara  tha  noat  relevant  references? 


2.  MOTIVATION  for  tha  aye tarn 

2.1.  PURPOSE:  What  la  tha  main  purpoee  of  tha  ayataa 
(••g.,  ganaral-purpoaa  computing,  raal-tlna  alr-trnfflc 
control,  a  tor«- and- forward)? 

2.2.  PHYSICAL  ENVIRONMENT!  Where  doaa  the  eyetem  operate 
(a  g.,  ground-baaad,  airborna,  apacabome)? 

2.3.  COMPUTING  ENVIRONMENT!  How  d">ee  tha  ayataa  relate 

computationally  to  lte  environment  (a.g.,  locally, 
remotely,  via  a  network,  interactively,  via  parlpharela. 
with  huaan  uaara)  ?  * 


2.4.  COMPUTING  OBJECTIVES!  What  ara  tha  epadflc  computing 
objectives,  regarding  capability,  capacity,  performance 
(throughput  or  raaponsa),  configuration  acaleablllty, 
raal-tlma  delays,  ate.  (aa  relevant)? 


2.5.  RELIABILITY  OBJECTIVES!  What  ara  tha  specific  ayataa 
reliability  objectives,  with  raapaet  to  daslrad 
availability  during  what  parlod,  minima  time  to  system 
failure,  aaxlmm  permitted  duration  of  outage,  etc.? 


2.6.  DYNAMIC  VARIABILITY:  How  may  these  objectives  very 
during  operation?  (E.g.,  how  may  performance  degrade? 
May  performance  ba  explicitly  exchanged  for  Increased 
reliability?) 


2.7.  PENALTIES:  whet  ara  the  penalties  arising  from 
faulty  operation?  (Poaelbla  oxampjae  Include  loaa  of 
life,  badly  dacraaaad  performance,  the  necessity  of  manual 
Intervention,  loaa  of  revenue,  etc.) 

2.8.  CONSTRAINTS:  Whet  explicit  physical  constralnta  exist 
(e.g.,  with  respect  to  elte,  weight,  power,  coat)? 

2.9.  TRADEOFFS:  What  critical  tradeoffs  exist  among  the 
objectives? 


3.  DESCRIPTION  of  the  eyetem 

3.1.  ARCHITECTURE 

3.1.1.  CONFIGURATIONS 

3.1.1.1.  INTERCONNECTIViTYi  What  la  the  basic  configura¬ 
tion,  and  whet  restrictions  exist  on  interconnaetivlty? 
(You  may  choose  to  Include  a  block  diagram,  a  PMS  diagram 
a  la  Ball  end  Newell,  or  other  useful  representation.) 

3. 1.1. 2.  RANGE:  Whet  Is  the  range  over  which  config¬ 
urations  era  sensible  (minimum  to  maxlaua),  e.g,,  how  many 
processors,  how  many  memory  modules  (of  what  alas  and  word 
length,  and  with  whet  restrictions  If  any),  etc.? 

3. 1.1. 3.  CAPABILITY:  What  la  the  effective  computlug  power 
of  tha  smallest  sensible  configuration  In  3,1,1.27  Please 
coapare  It  roughly  with  a  well-known  system  (e.g.,  360/40, 
65,  195),  and  elte  a  ball-park  figure  for  the  niober  of 
addition*  par  aacond.  Capability  required  for  fault-® 
tolerance  should  not  be  Included. 

3.1.2,  EXECUTIVE  and  operating  system 

3.1.2. 1.  MOMS  of  operation:  Hct#  dues  the  system  operate? 
(E.g., is  each  procasaor  multlprogrammabls?  Is  indapandent- 
procese  multiprocessing  possible?  Is  cooparatlve-procass 
multlprogrammed  multiprocessing  poesibls?) 

3. 1.2. 2,  SOFTWARE  organltatlon!  What  la  the  structure  of 
the  ayataa  software?  How  Is  it  distributed  with  raapaet 
to  tha  hsrd.'trs? 


3.2,2.  FAULTS  NOT  TOLERATED:  What  faults  eannot  bs 
tolerated  by  tha  system,  and  what  ara  tha  corresponding 
offsets?  idsntlfy  the  weakest  links. 


NOTE:  Faults  may  bs  chsrecterited  in  many  ways,  Including 
type  (a.g.,  faulty  hardware  at  various  levels  such  as  s 
chip,  nodule,  bus,  power  supply,  arithmetic  unit, 
processor,  memory;  faulty  software  such  as  in  tha 
executive,  In  a  compiler,  or  In  an  applications  program: 
-aulty  usage  and  bad  Inputs),  nature  (a.g.,  timing 
considerations,  old  age,  varloua  physical  phenomena), 
duration  and  frequency  (a.g.,  one-shot,  recurrent, 
permanent),  scope  (e.g,,  Isolated  faults,  correlated  or 
independent  multiple  faults,  with  varying  degrees  of 
propagation),  affect  (random,  predictable),  ate. 

3.2.3.  TECHNIQUES:  What  basic  techniques  are  employed  to 
provide  fault-tolerant  capability,  and  whan,  where,  and 
hew  ere  they  ussd?  Include  hardware  and  software 
techniques . 


NOTE!  Applicable  techniques  include  (possibly  In 
eosibinatlon)  replication  (e.g.,  triple-modular  redundancy 
at  various  levels,  redundant  computations  using 
Independent  algorithms),  coding  (i.g.,  error-detecting  or 
-correcting  codes  on  a  bus,  in  memory.  In  arithmetic), 
repetition  and  rollback,  reconfiguration  (including 
removal  without  replacement  and  replacement  with  spares), 
diagnostics  (e.g.,  stand-alone,  on-llna,  interactive; 
preventive,  emergency;  reiote,  local),  protection  (of 
processes,  data,  programs,  ate.),  and  outside  intervention 
(hinan  or  otherwise).  These  techniques  may  ba  used 
*6*?4cally  (a.g.,  always  inroked)  or  dynamically  (a.g., 
configured  as  needed) ;  at  v  irlous  module  levels  in 
hardware  and  software,  combination  with  certain  events 
and  with  certain  other  techniques. 

3.3.  NOVELTY:  What  ere  tha  most  unusual  design  fasturss  of 


3.4.  INFLUENCES:  What  other  efforts  (systems,  research) 
have  had  an  influence  on  your  system  design? 

3.5.  HARD-CORE:  If  there  la  a  concept  of  "hard-core"  In 
your  system,  what  Is  Its  significance?  (Plaaae  define 
your  concept.) 


4,  JUSTIFICATION  for  the  syatsm 

4.1  RELIABILITY  EVALUATION:  How  is  reliability  estimated 
and/or  demonstretsd  (a.g,,  via  analysis,  simulation, 
•tlmulation  of  fault*,  theoretical  argiaente)? 

4.2.  COMPLETENESS  OF  EVALUATION:  How  complete  la  your 
design  evaluation? 

4.3.  OVERHEAD:  What  parcantage(a)  of  total  system 
raaouresa  do  you  attribute  to  tha  achievement  of 
fault-tolerance?  (Consider  coat,  logic,  execution  time, 
memory,  etc.,  ea  applicable.) 

4.4.  APPLICABILITY:  What  la  tha  potential  ranee  of  applic¬ 
ability  beyond  that  stated  in  sections  2.1  -  2.4  above? 

4.5.  EXTENDABILITY:  In  what  ways  could  tha  ayataa  design 
ba  advantageously  extended,  with  what  increase  In  coat, 
and  to  what  affect? 

4.6.  CRITICALITIES:  Hew  critically  do  the  design  choices 
match  the  daeign  goal*?  (E.g.,  could  alight  changaa  in 
goals  result  in  greet  savingr  in  dealgn,  Implementation, 
and/or  operation?  la  multiprogramming  or  multiprocessing 
critical?  Ib  the  choice  of  hardware  critical?) 

4.7.  IMPLICATIONS :  What  apeclsl  requirements  (if  any)  does 
the  basic  daeign  Impose  (e.g.,  on  the  herdwere  designer., 
on  the  eoftwere  developers,  on  users  and  malntelnera) ? 


5.  CONCLUSIONS 

5.1.  STATUS:  What  la  tha  current  statue  of  tha  system? 

5.2.  EXPERIENCE:  Whet  tor  elusions  can  you  reach  based  on 
your  vxparlenea  with  the  ayataa  to  date  (a.g.,  In  design. 
Imp lamentation  and  operation)? 

5.3.  FUTURE:  What  la  planned  for  future  development  or  use 
of  the  system? 

5.4.  ADVANCES:  Whet  developments  (theoretical  or 
pvectical)  would  ba  desirable  for  significantly  advancing 
Ibe  state  of  tha  art  in  fault-tolerant  computing? 


3.2.  FAULT  TOU. RANCH 

3,2.1.  FAULTS  TOLERATED:  What  faults  ara  tolaratad  by  tha 
ayataa,  with  what  resulting  affacta  on  ayataa  behavior? 


6.  CCMMElfTS  (Please  Include  eny  comment#  on  your  eyetem 
on  thie  questionnaire,  etc.  which  you  would  like  to  add.’ 
Opinions,  prejudices  and  philosophies  ere  welcomed. 


SURVEY  OF  FAULT  TOLERANT  COMPUTER  SYSTEMS 
Algirdzs  Avizienl. 

UCLA  Computer  Science  Dept.,  Loe  Angelas,  CA  and 
Spacecraft  Computer  Section,  JPL,  Pasadena,  CA,  June  1972 

1.  IDENTIFICATION 

1.1  NAME:  JPL-STAR  (Self- Teetlng-And-Repel ring)  Computer 


3.  DESCRIPTION 

3. 1  ARCHITECTURE 
3.J.1  CONFIGURATIONS 

3. 1.1.1  INTERCONNECTIVITY:  See  Figure 

3. 1.1. 2  RANGE:  One  processor  of  each  class  (operating); 
16  memory  modules  of  4096  words  each  (maximum  operating 
memory) 


1.2  RESPONSIBILITY:  Spacecraft  Computer  Section, 
Aatrlonlcs  Division  of  ths  Jet  Propulsion  Laboratory, 
Pesadena,  California. 

1.3  SUPPORT:  NASA  -  Office  of  Advanced  Reseerch  and 
Technology  (via  JPL) 


3. 1.1.3  CAPABILITY:  500  KHz  maximum  clock  rate  and 
byte-aerial  operation  In  laboratory  model. 

3.1.2  EXECUTIVE 

3. 1.2.1  MODES:  Only  one  processor  operates  at  a  given 
time  (Single-processor  organisation) 


1.4  PARTICIPANTS:  A.  Avlzlenls,  D.  A.  Rennela,  J.  A. 
Rohr,  F.  P.  Mathur,  G.  C.  Gilley 

1.5  START:  1961 

1.6  COMPLETION:  Operational  -  Spring  1969  (laboratory 
model),  modifications  continue 

1.7.  BIBLIOGRAPHY: 

*1.  A.  Avlsisnia,  et  al.,  The  STAR  (Self-Testing  and 
Repairing)  Computer:  An  Investigation  of  the  theory  and 
practice  of  fault-tolerant  computer  design,  IEEE  Trans. 
Computer  C-20,  pp.  1312-1321  (November  1971). 

*2.  A.  Avlzlenie,  "Design  of  fault-tolerant  compute  a," 
FJCC,  pp.  733-743,  1967. 

*3.  A.  Avlilenls,  "An  experimental  self-repairing 
computer,"  Information  Processing,  IFIP,  Vol.  2,  pp. 
872-877,  1968. 

*4.  A.  Avlilenla,  F.  P.  Mathur,  D.  Rennela,  and  J.  A. 
Rohr,  "Automatic  maintenance  of  aerospace  computers  and 
spececraft  Information  and  control  systems,"  Proc.  AIAA 
Aerosp.  Comput.  Syat.  Conf.,  Paper  69-966,  pp.  1-11, 
September  8-10,  1969. 

*5.  A.  Avltlenla,  "Concurrent  diagnosis  of  arithmetic 
processors,"  Digest  of  the  1st  Annual  IEEE  Comput.  Conf., 
pp.  34-97,  1967. 

*6.  A.  Avlzlenls,  "Arithmetic  error  codes:  Coet  and 
effectiveness  studies  t'  r  application  In  digital  system 
design,"  IEEE  Trans.  Comp,  -20,,  pp.  1322-1331, Nov  1971. 

*7.  F.  P.  Mathur  end  A.  Avlzlenls,  "Rsllablllty  analysis 
and  architecture  of  a  hybrid-redundant  digital  syatem: 
Generalized  triple  modular  redundancy  with  self-repelr," 
SJCC,  pp.  375-383,  1970. 


3. 1.2. 2  SOFTWARE:  The  programming  subsystem  consists  of 
three  modules:  an  assembler,  a  loader,  and  s  functional 
simulator.  An  executive  program  facilitates  coordinated 
use  of  these  modules.  The  operating  subsystem  r.onelsts  of 
two  modules:  the  resident  executive  module  and  the 
applications  programs  module.  The  programming  subsystem 
has  been  Implemented  on  the  Unlvac  1108.  The  modules  of 
the  operating  system  of  the  STAR  computer  software  system 
consist  of  the  resident  executive  module  and  the 
application  module.  The  STAR  resident  executive  augments 
the  self  testing  snd  repairing  features  of  the  hardware  In 
addition  to  Its  normal  functions.  The  standard  features 
include  Interrupt  control,  input/output  proceeslng  and  Job 
scheduling.  Novel  features  Incorporated  due  to  the 
fault— to>>—  -  architecture  of  the  STAR  computer  Include 
a  "C°I  '■ability,  reconfiguration  processing, 

rollba.  ■,  and  diagnosis  of  faulty  units.  The 

cold  start  -ty  resets  the  hardware  and  software 

after  a  disaster  restart  aa  well  as  prior  to  an  Initial 
load.  Reconfiguration  processing  Is  required  for  memory 
replacement,  since  software  assistance  Is  required  to  load 
a  newly  activeted  memory  unit.  All  programs  running  on 
the  STAR  computer  require  rollback  (recovery)  points.  The 
resident  executive  provides  rollback  status  storage  and 
controls  svents  which  are  nonrepeatable ,  l.e.,  they  may 
not  occur  more  than  once  even  If  a  rollback  takes  place. 
Finally,  it  Implements  diagnosis  for  faulty  units  to 
determine  the  cause  and  extent  of  failures  for  possible 
reuse.  The  present  application  modules  Include  floating 
point  arithmetic  subroutines,  and  test  and  demonstration 
programs.  The  application  programs  that  will  be  required 
for  space  missions  are  a  part  of  the  TOPS  control  computer 
subsystem  project. 


3.2  FAULT  TOLERANCE 

3.2.1  FAULTS  TOLERATED:  The  principal  goal  of  the  design 
Is  to  attain  fault  tolerance  for  a  variety  of  faults: 
transient,  permanent,  random,  and  catastrophic. 


*8.  F.  P.  Mathur,  "On  reliability  modeling  and  enalysis  of 
ultrareliable  fault-tolaranr  digital  systems,"  IEEE  Trans. 
Comp.,  C-20,  pp.  1376-1382. 

*9.  G.  C,  Gilley,  "Automatic  maintenance  of  spacecraft 
systems  for  long-life,  deep-apace  missions, "  Ph.D. 
dissertation,  Dept.  Comput.  Scl.,  UCLA,  September  1970. 

*10.  F.  P.  Mathur,  "Rsllablllty  estimation  procedures  and 
CARE:  The  computer  aided  reliability  estimation  progrem," 
Jet  Propul.  Lab.  Quart.  Tech.  Rev.,  Vol  1,  October  1971. 


2.  MOTIVATION 

2.1  PURPOSE:  Experimental  laboratory  GP  machine;  suitable 
for  spacecraft  control 

2.2  PHYSICAL  ENVIRONMENT:  Laboretory  environment 

2.3  COMPUTING  ENVIRONMENT:  Local  1/0  facilities 

2.4  COMPUTING  08JECTIVES :  Capeble  of  automatically 
maintaining  an  unmanned  apacecreft 


3.2.2  FAULTS  NOT  TOLERATED:  (a)  Transients  at  a  rate 
higher  than  allowed  by  the  length  of  "rollback"  segments 
of  programs;  (b)  shorted  bus  wires  (isolators  are 
employed)  or  power  switch  "on"  failures. 

3.2.3  TECHNIQUES: 

*1.  All  machine  words  (data  and  instructions)  are  encoded 
in  error-detecting  codes  and  feult  detection  occurs 
concurrently  with  the  execution  of  the  programs, 

*2.  The  computer  is  divided  into  a  aet  of  replaceable 
functional  units  containing  their  own  instruction  decoders 
and  sequence  generators.  This  decentralisation  allows 
simple  fault-location  procedures  and  simplifies  system 
Interfaces, 

*3.  Fault  detection,  recovery  and  replacement  ate  carried 
c'tt  by  special-purpose  hardware.  In  the  case  of  memory 
dansge,  software  augments  the  recovery  hardware, 

'  ‘.  Transient  faults  are  identified  and  their  effects  are 
corrected  by  the  repetition  of  a  segment  of  the  current 
program;  permanent  faults  are  eliminated  by  the 
replacement  of  faulty  functional  units. 


2.5  RELIABILITY  OBJECTIVES:  100,000  hour  survival  with 
0,95  reliability;  tolerance  of  transient  faults;  outage 
for  recovery  below  50  msec. 

2.6  DYNAMIC  VARIABILITY:  Maximum  computing  powar  required 
at  end  of  mleslon 


*5,  The  replacement  is  implemented  by  power  switching: 
units  are  removed  by  turning  power  off  end  connected  by 
turning  power  on.  The  informetion  lines  of  all  units  are 
permanently  connected  to  the  buses  through  isolating 
circuits;  unpowered  units  produce  only  logic  "aero1 
outputs. 


2.7  PENALTIES:  None  for  lab  model;  loss  of  spacecraft 
for  flight  model 

2.8  CONSTRAINTS:  None  for  lab  modal;  for  the  flight  model 
tha  valght  cf  tha  subsystem  was  not  to  excaed  40  lb.  and 
the  power  consianption  was  not  to  ba  greeter  than  40  U, 

2.9  TRADEOFFS:  None 


*6.  The  error-detecting  codes  are  supplemented  by 
monitoring  circuits  which  serve  to  verify  the  proper 
aynchronitation  and  internal  operation  of  the  functional 
unite. 

*7,  The  "herd  core"  teat  and  repair  processor  (TARP)  is 
protacted  by  triplication  end  replacement  of  failed 
members  of  the  triplet. 


3.3  NOVELTY:  Power  twitching,  statue  signele,  encoding 
of  ine. .ructions,  emphasis  on  transient-recovery  with 
program  survival. 

3.*  INFLUENCES:  Theoretical  work  by  Reed  and  Brlmley; 
Kruus  and  Seshu;  Grlesmer,  Miller  and  Roth. 

3.5  HARD-CORE:  The  "hard  core"  monitor  of  the  STAR  system 
is  designated  as  TARP  (test  and  repair  processor)  in  the 
Figure.  Thj  TARP  monitors  the  operation  of  the  STAR 
computer  by  two  methods:  (1)  testing  every  word  sent  over 
the  two  data  buses  for  validity  of  its  code;  snd  (2) 
checking  the  status  messages  from  the  functlonsl  units  for 
predicted  responses. 

Three  fully  powered  copies  of  the  TARP  sre  opersted  at  all 
tl,»es  together  with  n  standby  spsres  (n  -  2  in  the  present 
design).  The  outputs  of  the  TARPs  sre  decided  by  a 
2-out-of-(n+3)  threshold  vote.  When  one  powered  TARP 
disagrees  with  the  other  two,  the  recovery  mode  is  entered 
and  an  attempt  is  made  to  set  the  internal  state  of  the 
disagreeing  unit  to  match  the  other  two  units.  If  this 
TARP  rollback  attempt  falls,  the  disagreeing  unit  is 
returned  to  the  standby  condition  snd  one  of  the  standby 
units  receives  power,  goes  through  the  TARP  rollback,  and 
joins  the  powered  triplet*  The  computer  is  now  restarted, 
a  rollback  performed,  and  standard  operation  continues. 
Because  of  the  three  unit  requirement,  design  effort  has 
been  concentrated  on  reducing  the  TARP  to  the  least 
possible  complexity.  Experience  with  the  present  model 
haa  led  to  several  refinements  of  the  design. 

The  replacement  of  faulty  functional  units  Is  commanded  by 
the  TARP  vote  and  is  Implemented  by  pc  iwitchlng.  It 
offers  several  advantages  over  the  swi:  :g  of 

informstlon  lines  which  connect  the  —  3  the  bus.  The 

number  of  switches  are  reduced  to  on  nit,  power  is 

conserved,  and  strong  Isolation  is  jd  for 

cstsstrophlc  failures.  Magnetic  po.  switches  have  been 
developed  which  are  part  o.  each  uni’  power  supply  and 
are  designed  to  open  for  most  lnten  failures.  The 
threshold  function  la  Inherent  in  the  control  windings  of 
the  switch.  The  information  lines  of  each  unit  are 
permanently  connected  to  the  buses  through 
component-redundant  isolation  circuits.  The  signal  on  a 
bus  Is  the  logic  OR  of  all  Inputs  from  the  units,  snd 
unpowered  units  produce  only  logic  zero  outputs.  The 
power  switch  snd  the  buses  utilize  component  redundancy 
for  protection  against  fatal  "ahorting"  failures. 

4.  JUSTIFICATION 

4.1  RELIABILITY  EVALUATION:  The  computing  operations  for 
the  analysis  was  done  with  the  aid  of  the  computer-aided 
reliability  estimation  (CARE)  program,  which  was  developed 
as  s  design  tool  during  the  reliability  study.  CARE  Is  s 
software  package  developed  on  the  Unlvac  1108.  CARE  may 
be  interactively  accessed  by  a  designer  from  a  teletype 
console  to  calculate  his  reliability  estimates.  The  input 
Is  in  the  form  of  s  syatem  configuration  description 
followed  by  queries  on  the  various  reliability  parameters 
of  interest  and  their  behavior  with  respect  to  mission 
time,  fault  coverage,  failure  rates,  dormancy  factors, 
allocated  spares,  and  partitioning.  The  CARE  program  is 
extensible,  snd  it  msy  be  updated  to  incorporate  new 
reliability  models  as  they  become  svailablc.  Physicsl 
fault-injection  experiments  are  currently  in  progress. 

4.2  COMPLETENESS  OF  EVALUATION:  Experiments  sre  expected 
to  continue  through  1972. 

4.3  OVERHEAD:  Depends  on  the  number  of  spares.  With  one 
spare  for  eacn  module— sbout  150  percent  extra  cost  (i.e. 

607  over'iH  id) .  * 

4.4  APPLICABILITY:  Various  real-time  applications  that 
require  very  fast  recovery. 

4.5  EXTENDABILITY :  Spare  proceaaors  could  be  utilized  in 
a  multiprocessor  mode.  Additional  buses  and  supervisory 
mechanisms  would  be  required* 

4.6  CRITICALITIES:  The  design  goal  was  a  better 
understanding  of  replacement  systems.  In  order  to  retain 
contact  with  the  practice  of  computer  design,  it  was 
decided  to  design  and  construct  an  experimental 
general-purpose  digital  computer  which  would  incorporate 
dynamic  redundancy  (i.e.,. fault  detection  and  replacement 
of  failed  subsystems)  as  integral  parts  of  its  structure. 

The  design  objectives  have  been  carried  out  and  the 
system,  called  the  STAR  computer,  began  operation  in  1969. 
The  modular  nature  of  the  STAR  computer  hae  allowed 
systematic  expansion  and  modifications  that  are  still 
being  continued. 


An  early  objective  of  the  design  is  to  study  the  class  of 
problems  which  sre  encountered  in  transforming  the 
theoretical  model  of  s  self-repairing  system  into  s 
working  computer.  State-of-the  art  Integrated  circuit  and 
memory  technology  was  employed  in  the  design.  This 
objective  sppesrs  to  hsve  been  attained  reasonably  wrll. 

4.7  IMPLICATIONS:  Designers  must  give  (s)  advance 
attention  to  modularization  and  coded  operands;  (b) 
special  software  features  sre  needed  (see  3. 1.2. 2);  (c) 
users  must  observe  "rollback"  rules  lr.  programming. 

5.  CONCLUSIONS 

5.1  STATUS:  Operating  in  laboratory;  being  extensively 
tested  snd  modified  to  Improve  weaknesses  thtt  are 
uncovered. 

5.2  EXPERIENCE:  Practical  implementation  of  replacement 
systems  is  feasible.  Transient  faults  can  be 
systematically  eliminated  without  programs:  loss. 

Transient  tolerance  can  be  specified  in  terms  of 

duration"  snd  "frequency"  parameters. 

5.3  FUTURE:  The  research  and  development  program  which 
led  to  the  STAR  computer  is  continuing  in  several 
directiona.  The  design  of  several  improved  .second 
generation  STAR  functional  units  is  under  way,  including  s 
new  arithmetic  processor,  s  control  processor  for 
medium-scale  integrsted-circuit  implementation,  snd  the 
shared  READ-WRITE  memory  unit  for  the  storage  of  automatic 
maintenance  information  from  the  spacecraft  telemetry 
system.  Analysis  of  automatic  maintenance  algorithms  and 
design  of  a  command/data  bus  for  their  implementation  are 
under  intensive  study.  Other  current  investigations  sre 
concerned  with  the  following  areas:  (1)  hardware-software 
interaction  in  a  fault-tolerant  system  with  recovery, 
especially  the  interaction  of  the  TARP  snd  the  operating 
system;  (2)  studies  of  advanced  recovery  techniques,  I.e., 
poat-cstastrophic  restart,  TARP  replacement  schemes, 
recovery  from  massive  Interference,  partial  utilization  of 
failed  units;  (3)  advanced  component  technology, 
especially  methods  to  attain  bus  and  power  switch  (i.e., 
hard  core)  immunity  to  faults;  (4)  heuristic  studies’of* 
fault  tolerance  by  Interpretation  of  extensive  experiments 
with  the  STAR  breadboard  as  the  Instrument;  (5)  design  of 

a  second-generation  STAR-type  computer  with  universal 
processor  snd  stnrage  modules,  snd  their  implementation  by 
large-scale  integration;  (6)  computational  utilization  of 
the  spare  units  for  supplemental  tasks  in  s 
multiprocessing  mode. 

5.4  ADVANCES:  (a)  methods  of  coverage  measurement;  (b) 
technology  ad  cnees  in  isolator  snd  switch  design;  (c) 
studies  in  restart  ("roll-bsck")  Implementation  by 
automatic  methods. 

6.  COMMENTS:  Design,  construction,  and  testing  of 
laboratory  models  is  critically  Important  to  advance  the 
state  of  the  art  snd  to  gain  acceptance  among 
practitioners  of  design  in  Industry. 


COP  Control  prr.essor,  contains  the  location  counter  and 
index  registers. 

LOP  Logic  processor,  (two  copies  are  powered). 

MAP  Main  arithmetic  processor. 

ROM  READ-ONLY  memory,  16,384  permanently  stored  words. 
RWM  READ-WRITE  memory  unit  (4096  words,  two  copies 
powered,  12  units  directly  addressable.). 

IOP  Input/Output  processor,  contains  1/0  buffer. 

1RP  Interrupt  processor,  handles  interrupt  request, 

TARP  Test  and  repair  processor,  (three  copies  powered). 


SURVEY  OF  FAULT-TOLERANT  COMPUTING  SYSTEMS 
W.  c.  Carter 

IBM  Thomas  J.  Wataon  Resaarch  Center 
Yorktovn  Helghta  NY  10598 


1.  IDENTIFICATION 

1.1.  NAME:  I  am  reporting  mainly  on  a  long-term  research 
effort  in  techniques  for  fault-tolerant  computer 
architecture.  The  relevenf  prior  publications  heve  used, 
for  example,  the  terms  "mod-ilar  architecture", 
"self-repairing  computers",  'dynamic  checking",  "fault 
diegnosis",  "stand-by  sparing"  or  "dynamic  recovery"  in 
the  titles  and  the  authors  heve  been  some  subset  of  the 
participants  named  in  1.4.  For  present  purposes  I  will 
talk  about  a  paper  Moduler  Digital  Computer  system  called 
MDC  whose  principal  properties  will  be  specified  later. 
For  reality,  some  requirements  will  be  imposed  which  have 
nothing  to  do  with  fault  tolerance  per  se.  This  system 
does  not  really  exist,  and  will  not  exist,  but  is 
specified  to  provide  a  focus  for  our  fault  tolerant 
computing  research. 

1.2  RESPONSIBILITY:  IBM  Research. 

1.3  SUPPORT:  Support  haa  come  from  IBM,  U.  S.  Air  Force 
and  NASA. 

1.4  PARTICIPANTS:  W.  G.  Bouricius,  W.  C.  Carter,  E.  P. 
Hsieh,  D.  c.  Jeasep,  Jr.,  G.  P.  Putzolu,  J.  P.  Roth,  P.  R. 
Schneider,  C.  J.  Tan,  A.  B.  Wadia. 

1.5  START:  Formal  initiation  occurred  in  March,  1966. 

1.6  COMPLETION  Open  ended.  No  end  item  is  scheduled. 

1.7  BIBLIOGRAPHY: 
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B, ,  McCarthy,  c.  E.,  Milligan,  F.  G.,  "Design  techniques 
for  MARCS"  (Modular  Architecture  for  Reliable  Computet 
Systems),  NASA  Contract  NAS8-24883,  RA12,  IBM  T.  J.  Watson 
Research  Center,  Report  Number  70-208-002.  March  26,  1970. 

‘Carter,  W.  c. ,  Jeasep,  D.  C.,  Wedia,  A.  8.,  Schneider,  P. 
R. ,  Bouricius,  W.  G.,  "Logic  design  for  dynamic  and 
interactive  recovery",  IEEE  TC,  Vol.  C-20,  Nov.  1971. 

2.  MOTIVATION 

2.1  PURPOSE:  Real  time  control,  data  acquisition  and  data 
management. 

2.2  PHYSICAL  ENVIRONMENT:  Aerospace  applications  have 
predominated  in  specific  design  decisions.  Modularity 
should  ensure  wide  applicability. 


2.3  COMPUTING  ENVIRONMENT:  The  -DC  is  planned  to  be  able 
to  run  the  gamut  from  being  Insulated  from  human  control, 
oerving  s  variety  of  sensors  and  effectors,  to  being  able 
to  accept  ground-based  human  directed  control. 

2.4  COMPUTING  OBJECTIVES:  Predicted  configuration 
acaleebility  primarily  under  internal  control  including 
systems  which  sre  fault  tolerant  by  masking  redundancy,  by 
stand-by  redundancy,  or  by  software  checks;  systems  whose 
use  of  power  is  variable  (but  whose  thruput  is  affected); 
and  systems  operating  in  parallel.  The  major  objective  is 
to  provide  means  for  meeting  various  requirements  with  a 
high  degree  of  confidence. 

2.5  RELIABILITY  OBJECTIVES:  The  system  is  to  be  designed 
to  meet  varying  specific  mission  reliability  objectives 
with  a  high  degree  of  certainty.  Examples  sre  survival 
for  n  years  with  a  probability  p;  "Fail  operational,  fail 
operational,  fail  safe",  or  reliability  variable  with 
mission  task. 

2.6  DYNAMIC  VARIABILITY:  As  stated  above,  dynamic 
variation  of  system  parameters  such  ss  performance, 
reliability  and  power  consumption  with  confidence  in  the 
design  as  a  major  objective. 

2  1  PENALTIES:  Variable  with  mission,  ranging  from  loss 
of  human  life  through  expensive  flight  hardware  to 
abortion  of  flight  objectives. 

2.8  CONSTRAINTS:  Hardware  must  be  designed  to  fit  weight, 
power  and  aize  requirements,  yet  able  tc  have  thruput 
compatible  with  mission  requirements  and  to  support  the 
software  necessary  for  reasonable  programming  effort  per 
mission. 

2.9  TRADEOFFS:  Hardusre  efficiency  and  potential  thruput 
are  traded  for  1)  system  reliability  as  defined  per 
mission  phase;  2)  simplification  of  recovery  process  snd 
other  basic  executive  functions;  3)  high  malfunction 
coverage  and  design  certification;  A)  ease  of  program 
validation;  5)  convenience  of  programming  and  ease  of 
diagnosis  for  external  equipment;  6)  aystem  flexibility. 


3.  DESCRIPTION 
3.1  ARCHITECTURE 

3.1.1.  CONFIGURATIONS 

3. 1.1.1.  INTERCONNECTIVITY :  Hie  basic  uniprocessor 
configuration  consists  of  partitioned  computer  subunits 
attached  to  several  busses.  The  basic  subunits  are  (see 
attached  rough  diagram):  ALU,  Scatch  and  Program  Control 
Unit,  Bus  Control,  I/O  Processor  and  Recovery  Control 
Unit.  The  bts  orientation  remains,  but  the  units  may  be 
modified  (microprogrammed)  for  varying  missions.  The 
system  consists  of  replicas  of  the  basic  subunits,  with 
configuration  control  governed  by  the  RCU  and  Executive 
Program.  A  major  problem  is  the  interface  design  to  meet 
the  constraints  of  fault  tolerance,  long  life,  end  varying 
modes  of  operation.  The  memory  is  encoded  with  a 
b-adjacent  error  correcting  code  and  spare  b  wide  subunits 
per  basic  module. 
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3.1. 1.2  RANGE:  the  range  of  the  system  is  nor  frozen  in 
f  ;hJt«tur«1  concept.  After  four  processors  the  lsw 
ol  diminishing  returns  sets  in  sharply  and  further 
partitioning  nay  well  be  s  better  bet  for  long  life.  The 
memory  will  consist  of  modules,  esch  module  consisting  of 
b-wlde  units  with  b-adjacent  coding  and  spare  b-wldth 
units.  The  upper  limit  depends  upon  the  hardware 
available,  but  hardware  does  not  appesr  to  be  critical. 

3. 1. 1.3.  CAPABILITY:  The  order  of  10E5  to  10E6  sddltions 
per  second  per  basic  system  with  a  minimum  of  256K-512K  32- 
bit  words  of  meaory.  I/O  will  be  handled  by  up  to  4  16- 
blt  parallel  c.iannels  with  50,000  transfers  per  second 
slmultan-  "*’y  on  one  input  and  one  output  c.annel.  The 
I/O  processor  will  handle  the  details  of  l/n  control  under 
direction  from  the  processor  Executuve. 


3.1,2  EXECUTIVE:  Tne  standard  executive  control 
(allocation,  scheduling,  dispatching,  I/O)  will  be 
achieved  by  replicated  software  routines.  These  tsaks 
have  not  been  studied  much. 


3, 1.2,1,  MODES  OF  OPERATION:  Each  processor  is 
wultiprogrammable ,  System  operation  includes  fault 
masking,  multiprocessing  with  hardware  fault  detection  and 
multiprocessing  with  uofcwsre  snslysis.  The  mode  of 
operation  of  most  concern  is  that  of  recovery  initiation, 
the  interaction  of  the  recovery  and  error  snslysis 
programs  of  the  executive  end  the  RCU.  Recovery  and  audit 
programs  always  run  background  whether  the  system  is  in 
fault  masking,  fault  detection  or  software  analysis  modea. 


3. 1.2.2,  SOFTWARE  ORGANIZATION:  The  syatem  software  will 
be  distributed  among  the  processors  and  analyzed  by  audit 
routines  for  early  detection  of  c:rors. 

3.2.  FAULT  TOLERANCE 

3,2.1  FAULTS  TOLERATE"):  In  the  error-masking  mode,  any 
number  of  faults  which  affect  only  one  partitioned 
sub-unit  ca:  be  tolerated.  The  system  handles  transient 
faults  with  instruction  retry  or  permanent  faults  with 
hardware  controlled  reconfiguration.  The  cause  is 
irrelevant  as  long  as  the  interface  detects  disagreement. 
The  disagreement  circuits  are  self-checking  so  faults  in 
t  em  are  detected.  Initially  the  same  malfunction  in 
three  units  ia  necessary  to  defect  the  system.  After 
reconfigurations  two  faulty  units  may  escape  detection. 

In  the  error  detection  mode,  faults  causing  s  single 
subunit  to  be  in  error  are  detected.  At  this  point  the 
asme  errors  in  two  unita  will  be  undetected.  Diagnosis 
and  software  recovery  is  necessary  for  continuation. 


Faults  detected  by  software  checks  are  detected  snd 
recovery  should  follow  in  the  unchecked  multiprocessing 
mode.  Faulty  software  may  be  detected  by  the  RCU  time-out 
tests  and  system  evaluation  procedures, 

3.2.2.  TECHNIQUES:  In  hardware  fault  tolerant  mode  the 
ayatem  ahould  F0  -  F0  -  FS  for  each  one  of  the  partition, 
of  the  ayatem  If  four  coplea  of  the  baalc  computer  are 
uaed,  Dlagnoala  can  continue  the  computation  with  one 
partition  unchecked.  Detailed  fault  nnslysia  rauat  be 
performed  to  validate  ouch  goala.  In  hardware  fault 
detection  moda  the  system  ahould  run  at  least  two 
multlprocesaor  hardware  checked  ayatema.  A  fault  would  be 
detected,  and  diagnoses  would  allow  continuation  with  one 
partition  unchecked  by  hardware.  Achieving  such 
hardware/flrmware/dlagnoels  goela  depends  upon  the 
development  of  many  tools  of  fault  anal] ala.  The  memory 
encoding  la  b-adjacent  multiple  error  correcting  and/or 
multiple  b-sdjacant  error  detecting.  The  codes  used  are 
variants  of  Reed-Solomon  codes  with  combinational  self¬ 
checking  tranalatora  which  pass  only  correct  code  words. 
Standard  alngle  Instruction  retry  la  available. 

Mlcrodlegnostlca  under  executive  program  control  with 
program  variable  Input  patterns  will  be  used  for  fault 
analysis.  The  executive  software  will  use  the  standard 
fault  tolerant  techniques  -  two  way  lista  with  pointer 
ver^*cahi°n  before  proceeding,  scored  data  and  programs 
will  be  taggec  with  redundant  Identification,  read  only 
programs  will  How  simple  updating  etc.  Rollback  t  <d 
restart  will  be  uaed  for  multi-proceaeing  with  hardware  or 
aoftware  error  detection.  The  RCU  monitors  constantly  for 
catastrophic  faults  —  those  not  detected  by  the  hardware 
and  software  testa.  The  standard  time-out  teats  snd 
system  performance  evaluation  routines  are  run  and 
controlled  by  the  RCU,  Power  Is  conserved  under  program 
control  by  forcing  n  cycles  between  memory  accesses, 

Imposed  by  a  counter  with  program  changeebla  centents. 


3.3  NOVELTY:  Reconfiguration  undar  hardware  control  in 
fault  masking  mode.  Choice  of  computer  fault  masking, 
multiprocessing  with  fault  masking  snd  various  forms  of 
detection,  multiprocessing  with  hardware  error  detection 
by  comparison,  multiprocessing  with  aoftware  error 
detection.  Storage  reliability  b>  b-sdjacent  multiple 
error  detecting  and  correcting  codes.  Seif  checking 
memopr  translators,  checking  circuits,  and  error-anslysis 
ciruits.  Use  of  power  under  program  control. 


_  ,  ,  .  .  ‘  -  total  eiiort;  Z,  S Hi 

Techniques  for  the  Realization  of  Ultra-Reliable 
Spacebome  Computers;  3.  MIT  -Draper  Leb.  lor  apaceborne 
multiprocessors;  4,  Rapid  emergence  of  LSI  for  feaslbllit 
of  much  redundant  hardware. 


3,.'  HARD-CORE:  Assuming  thst  hard  core  means  hardware, 
redundant  or  not,  whose  failure  will  produce  ^-.detected 
errors,  there  is  no  such  hardware  In  this  system. 
Hopefully,  the  software  can  be  validated  so  thst  equal 
claims  can  be  made  for  it. 


4.  JUSTIFICATION  FOR  THE  SYSTEM 

4.1  RELIABILITY  EVALUATION:  Architectural  reliability 
evaluation  by  Interactive  program  using  exponential 
failure  assumption  for  the  units.  Determination  of 
component  failure  rates  by  anslysla  baaed  upon  previus 
data,  experience,  and  analysis.  Logic  fault  analysis  of 
circuits  in  design  stage  by  interactive  fault  simulation 
programs.  Diagnostic  pattern  evaluation  by  simulation 
programs.  Memory  fsllure  predictions  by  careful 
probabilistic  fault  analysis  to  predict  error  patterns 
programmed  computation  of  the  circuit  failure  constants, 
programmed  evaluation  of  reliability.  Programmed 
analysis  of  RCU  functions.  Theoretlcsl  analysis  of 
design,  with  hardware  and  software,  in  complicated 
situations  (guided  by  simulation). 


COMPLETENESS  OF  EVALUATION:  Major  unsolved  problem. 

4.3  OVERHEAD:  Variable.  In  the  processors  about  a  3  1/2 
:1  logic  count  penalty  la  paid  (the  cost  la  much  less), 
n  the  memory  about  a  3!2  storage  penalty  is  paid.  In  the 
aoftware  the  coat  la  unknown,  but  considerable. 


4.4  APPLICABILITY:  The  concepts  can  be  uaed  elsewhere 
the  system  la  oriented  toward  space  and  extremely  high 
reliability  applications. 


4.5  EXTENDABILITY:  This  computer  la  too  reliable  to  fit 
Into  most  other  ayatema.  For  extension  some  of  the  fault 
tolerant  techniques  in  the  computer  must  be  essed  for 
better  totsl  system  bslance. 


4.6  CRITICALITIES:  Multitasking,  aa  with  all  Executive- 
controlled  recovery  systems,  la  critical,  achieved  here 
with  multiprogramming.  Multiprocessing  la  an  Imposed 
condition,  but  small  system  simplifications  would  result 
if  this  condition  were  relaxed.  Design  validation  tools 
sre  criticsl. 


A. 7  IMPLICATIONS:  Architects  must  perform  automated  error 
and  recovery  analyais  while  doing  system  specification. 
Human  analysis  la  too  fallible.  Hardware  designers  must 
have  snd  use  tools  to  do  fault  analysis  aa  they  design. 
After  the  first  pass  they  must  do  design  validation  and 
lterat'-.  Software  designers  must  participate  In  the 
Initial  decisions,  must  produce  more  techniques  for 
producing  self-checking  programs,  and  must  produce  the 
tools  for  progrsm  validation.  Applications  programmers 
must  validate  their  programs  (top  down  programming 
techniques  will  help),  and  must  follow  system  rules  (not 
so  far  known). 


5.  CONCLUSIONS 

5.1  STATUS:  This  system  la  the  collection  of  a  group  of 
ideas  from  a  research  project. 

5.2  EXPERIENCE:  None  to  report  to  date. 


3,3  FUTURE:  The  system  will  be 
form  as  a  peper  study  only. 


pursued  only  in  a  modified 
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software  -  will  provide  many  a  bottleneck  for  fault 
tolerant  computing.  The  bealc  problem  of  definition  o 
tault  tolerant  computing  will  be  with  us  -  do  we  const, 
any  algorithm,  procedure? 
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X.  IDENTIFICATION 

!*P°rtlD‘  °°  •  lon«-Mra  development 
V“  !'»P°rt*<!  br  <Uff*M0t  project*  at 
**£??*?“*  **■■■•  T*1*  foilwring  title*  hare  baas  wad  for 
pibllahad  report*!  Ior 

*  "A  Feult-Tolere-t  Information  Processing  System  for 
Advsmcad  Control,  Suldamca.and  Navigation". 

*  "Spate  Trane portation  Syatea  Data  Management  Syatam". 

In  addition,  an  experiaentrl  thraa-procaaaor 
thraj-acrattbpad  brawlboard  hu  bean  given  the  acronym 
C1XBERUS  for  the  thraa-haadad  dog  In  elaealeal  mythology , 
The  acronym  engendered  tha  title i  Controlled  Brror 
Kacoyery  Beheelor  Employing  Redundant  Uaa  of  Seratchpada. 
In  what  follow*,  I  uaa  "the  aye tarn"  to  naan  tha  ganaral 
concept,  rather  than  a  apadflc  hardware  daalgn,  Thla 
glwaa  me  tha  adwantaga  of  being  able  to  ba  light  on  my 
feat  and  adept  to  any  altuatlcn  bafora  tha  fact. 

1.2,  RESPONSIBILITY!  Thla  work  la  In  tha  Digital 

D*T?i0,Tnt  Group  of  th*  Stark  Draper  Laboratory, 

a  dlvialon  of  M.I.T. 


Jhf'wae^0*1  S0™C,SI  So  far  •!!  a  up  port  ha*  coma  from 
tha  NASA  Manned  Spacecraft  Center. 


l.A.  PARTICIPANTS!  MIT  and  N, SA/MSC. 

1.5.  START!  Work  In  thla  area  began  In  1966. 

1.6.  COMPLETION!  Open  ended.  No  and  item  la  a  chad-ala  4 . 

1.7.  BIBLIOGRAPHY: 


Rr  L"  Aion*°«  A*  L*  Hopklna ,  Jr.,  and  H.  A.  Thaler, 
Daalgn  Criteria  for  a  Spacecraft  Computer",  Spacaborne 
Multlorocaaelnp  Seminar,  pp.  23-28,  NASA  ERC,  Boaton 
Museum  of  Science,  Oct.  1966. 


R.  L.  Aloneo,  A.  L.  Hopklna,  Jr.,  and  H.  A.  Thaler,  "A 
Multlproceaalng  Structure",  Difeat  of  the  Flret  Annual 
IEEE  Computer  Conf.,  pp.  56-59,  Chicago,  Sapt.  1967. 


*  A.  I.  Croon  at  al.,  "STS  Data  Managamant  Syatam 
Daalgn  ,  MIT  C.S.  Draper  Laboratory,  Cambridge,  Me**.. 
Report  E-2529,  Juno  1970.  *  ’ 


*  A.  L.  Hopklna,  Jr.,  "A  Fault-Tolerant  Information 
Procaaalng  Concept  for  Space  Vehicle*",  IEEE  True. 
Computer*.  Vol.  C-20,  pp.  139A-1403,  Now.  1971. 


2.  MOTIVATION 

2.1.  PURPOSE:  Real  time  control,  data  acquleltlon  and 

data  managamant. 

2.2.  PHYSICAL  ENVIRONMENT!  In  principle  It  could  ba  any, 
but  aaroapac*  application*  haw*  predominated  In  daalgn 
dadalona. 

2.3.  COMPUTING  ENVIXOMtENT  i  Syatam*  conaldarad  her*  ar* 
envisioned  a*  largely  aalf-contained  Information 
procaaalng  ay a torn*  earring  a  variety  of  aanaora  and 
effector*  Including  htaum  operator*.  Such  ayataw  would 
b*  distributed,  hierarchical  and  radwdant.  Central 
fault-tolarant  multiprocessor*  would  coMunleat*  over 

<*•!•  bum**  to  local  procaaaor  coaplazaa  adaddad  in 
aubayatama  of  th*  total  syatam.  A  principal  application 
conaldarad  for  thla  approach  waa  tha  Spaca  Shuttle,  whore 
tht  Orblttr  would  hm  ouo  control  *ult  i  pro  com  or  with 
adequate  redundancy  and  spar*  hardware  to  b*  operational 
after  three  ma  If  wet  Iona,  Each  subsystem  or  grow  of 
Identical  aubayatama  would  ba  served  by  alngla  or 
radwdant  local  proaamaora,  aa  appropriate,  to  fulfill  tha 
redundancy  requirement  for  that  aubayaram  or  grow. 

Tha  Booatar  stag*  of  th*  Spaca  Shuttle  would,  In  this 
concept  contdn  a  ay  a  tea  a  ini  Ur  to  that  of  th*  Orb  iter, 
capable  of  Cornwall  cat  ii*  with  It  by  way  of  a  serial  bia 
compacting  tha  two  central  multlprocaaro.i.  All 
rrmawl cation  between  a  central  multiprocessor  and  Its 
local  procaaaor*  would  ba  via  a  aerial  data  bw. 


2.6,  COMPUTING  OBJECTIVES  FOE  THE  CENTRAL  MULTIPROCESSOR: 
Variable  froa  th*  order  of  10E5  (l.o.,  10  to  tha  5)  to  tha 
order  of  10E6  operatlona  per  second,  with  namcry 
capacities  of  from  2E16  to  2E17  words  of  wain  random 
•ecaaa  aanory.  Input-output  bandwidth  10E5  useful 
blta/aec  on  a  10E6  puUe-per-eecond  bus.  Reaction  tine 
order  of  10  Billiseconds. 


.  — unum-rivas!  various  oojacciva*. 

One  example  lo  airline  applications  whar*  less  than  one 
cats* trophic  syatam  nalfwctlon  in  10E7  flight*  la  sought. 
Other  objectives  are  stated  In  terms  of  tha  nu*er  of 
Individual  malfunctlona  which  cam  ba  tolerated  la  a 
fUyht,  such  aa  "Pall  operational,  fall  operational, 
failsafe  (PO-PO-FS).  Thu  syatam  la  tanarally  meant  to  ba 
used  in  vary  high  reliability  applications. 


r,6,  DYNAMIC  VARIABILITY:  Graceful  degradation  lo 
available  aa  a  naans  of  exchanging  performance  for 
reliability. 


2.7.  PENALTIES!  In  th*  Spaca  Shuttle  application,  as  In 
possible  aircraft  application*,  hunan  Ufa  la  concerned 
aa  wall  aa  expensive  flight  hardware. 

2.8.  CONSTRAINTS!  In  Spaca  Shuttle  amd  aircraft, 
approximately  2  cubic  fast,  120  lb.,  300  watt*. 
(Estimate  for  a  central  multiprocessor) • 

2.9.  TRADEOFFS!  Hardwara  efficiency  1*  traded  for  1) 
•Fafam  reliability,  2)  high  nalfwctlon  coverage,  3)  a  as  a 
of  program  verification,  41  system  flexibility. 

The  PuJtar  of  faults  tolerated  la  variable  through  a 
combination  of  replication  and  sparing.  Procaaaor*  and 
namorlaa  can  ba  added  (delated)  to  lncraoao  (decrease) 
processing  amd  memory  resource#. 


3.  DESCRIPTION  OF  THE  SYSTEM 

3.1.  ARCHITECTURE 

3.1.1.  CONFIGURATIONS 

3.1.1.1.  INTERCONNECT! VI TY i  The  syatam  makes  ext  ana 1  vs 
W*  of  rapUcatlon,  and  consequently  connections  have  e 
high  coat.  Serial  and  byta-aarlal  buses  are  used  batwaan 
baale  unit*.  Multiplexer*  era  employed  to  prevent  alngla 
wit  malfunction*  from  spreading  to  all  copies  of  a 
radix  dent  bus.  Tha  canonical  Interconnection  scheme  i* 
shewn  In  Figure  1. 

3. 1.1. 2.  RANGE:  No  range  limits  have  baan  determined,  but 
th*  following  n where  nay  ba  typical  for  an  aaroapaca 
application.  Thors  ara  two  currant  compositive 
concaptualltatlona  of  th*  ayatam.  Thaaa  nwbara  rapraaent 
fl>a  nmear  and  laaa  wall  developed  concept. 

*  6«  Number  of  simultaneous  job  otopa  In  process 

*  Dagra*  of  replication  of  cash  processor-scratchpad 

*  >  Number  of  apara  processor-* cretchpeda 
*21-  Total  procaaaor  ocratchpads  -6x3+3 

*  4-  Niabar  of  Independent  memory  blocks  of  16K 

*  3-  Dagra*  of  replication  of  each  block 

*  3-  h  nber  of  opera  blocks 

*15-  Tu  memory  block  modulo*  -4x3+3 

Th#  numbs,  of  proceatot-ecratchped#  and  memory  blocks  can 
be  Increased  up  to  th*  practical  bandwidth  limit  of  tha 
proesaaor-manory  bus  and  tha  I/O  bus. 

3. 1.1. 3.  CAPABILITY!  Th*  order  of  10E5  to  10E6  addition* 
par  second  and  tha  order  of  2E14  word#  of  memory.  Three 
processors  would  ba  tha  anallast  "sensible"  number. 

3.1.2.  EZBCUTIVE 

3.1.2. 1.  MODES  OP  OPERATION!  All  programs  are  segmented 
Into  job  otopa  which  ar*  dispatched  by  a  floating  form  of 
executive.  Each  job  otap  occupies  one  procaaaor  full  time 
uhilo  It  twa.  Multiprocessing  1*  th*  normal  operating 
mods.  Multlprograwlng  of  each  procaaaor  la  not 
anvlalonad. 

3. 1.2. 2.  SOFTWARE  ORGANIZATION!  I/O  procaaalng  la 
queal.. dedicated  to  ona  procaaaor  (l.a.  It  cm  float  but 
doaa  aa  only  when  malfunction  makes  it  necessary). 
Executive,  monitor,  and  reconfiguration  programs  ar*  rw 
on  an  aa-naadad  basis  by  each  procaaaor  aa  it  finish**  a 
Job  atop. 


3,2,  FAULT  TO  LX  RAN  Ck 

3.2.1.  FAULTS  TOLERATED:  Individual  unit*  (e.g. 
prouuor,  memory  'tilt ,  multiplexer)  can  malfwctlon  cm* 

•t  *  tira  with  no  restriction  on  whet  the  nature  of  the 
anlfunctlon  1*.  Irror*  ere  nesked  by  the  system  until  it 
reconflfuree  Itself  to  a  feult-telerent  state. 

3.2.2.  FAULTS  NOT  TOLERATED :  Cartels  malfunction  pairs 
which  occur  elaultenaously  or  close  together  in  tin*  can 
produce  loss  of  date  end  nany  require  *  progras  restart. 
Incorrect  specification*  or  progras  nelfinctlon*  can 
defeat  the  eysten.  Systematic  hardware  melfunctlons  In 
which  the  sea*  malfunction  occurs  In  two  redundant  tnlta 
can  defeat  the  system. 

3.2.3.  TECHNIQUES :  Two  different  concepts. 

Flret  concapti  *11  proceeaoru  ere  duplexed  for  detection. 
All  scratchpad*  ere  trlplaxed  for  masked  dwp  capability. 
Single  Instruction  restart.  Graceful  degradation  of 
processor-scratchpad  group*.  Triplex  memory  unit*  with 
dedicated  spare*.  Triplex  bueaa  with  spares. 

Multiplexer*  Isolate  buses  from  failed  greupe  of  wit*. 

Second  concept:  processor-scratchpad  units  an  organised 
Into  group*  of  three  under  software  control.  Each  looks 
for  dlsagreeaent.  If  dlsagreanant  occurs,  continus 
running  to  end  of  job  step,  then  enter  reconfiguration 
progrms.  Graceful  da gradation  of  Individual 
processor-scratchpad  unit*  (rather  than  groups  of  three 
scratchpad*  end  two  processor*  a a  In  flret  concept). 
Triplex  memory  units  with  noa-dadlceted  spsrts.  Triplex 
buss*  with  spa.*.,  Multiplexer*  Isolate  buses  from  failed 
Individual  unit*  (re thaw  than  groups  as  in  flret  concept). 

In  both  concept*,  software  configuration  control  1*  used, 
which  Is  valid  es  long  a*  *  working  processor  group, 
memory  group,  ad  bus-multlplaxar  group  are  available. 
Multiplexers  participate  in  configuration  control. 

3.3.  »  'VELTT:  Single  Instruction  restart.  Absence  of 
interrupt*  and  program  rollbacks.  Distributed  monitor  and 
reconfiguration  fwctlons.  lisa  of  multiplexers  to  isolate 
bus  end  unit  msl fwctlons.  Fault- tolarant  clock. 
Hierarchical  system  with  fault  tolerance  extended  Into 
aubeyetmea. 

3.4.  INFLUENCES:  Rapid  emergence  of  LSI  memories  end 
processor*  he*  encouraged  use  of  replication  and 
partitioning  with  simple.  Identical  wit*.  Apollo 
Guidance  Computer  Experience  prompted  elimination  of 
Interrupt*  and  roUbeu'..  for  the  sake  of  program 
verification.  Carter  and  Bourldua  for  rail  ability 
models.  Avltlenl*  for  concepts  of  fault  tolerance. 

3.5.  HARD  CORE:  Assuming  that  herd  cor*  means 
nou-radwdsut  hardware,  there  1*  no  herd  cor*  In  thla 
system.  Configuration  control  la  *  software  function 
using  the  available  hardware  to  configure  the  system. 


4.  JUSTIFICATION 

1.1.  FELIABILITT  EVALUATION:  So  far  mostly  geared  toward 
rO-PO-FS.  Soma  Probabilistic  analysis.  No  reliability 
projection*  a*  yet  elnca  hardware  has  not  bean  eelacted 
and  failure  rates  ere  therefor*  not  '  town. 

4.2.  COMPLETENESS  OF  EVALUATION:  Herowars  not  sslac:*d, 
hence  failure  ret*  not  kr.own. 

4.3.  OVERHEAD:  About  SOT  of  the  system  is  devoted  to  the 
achievement  of  fault  tolerance. 

4.4.  APPLICABILITY :  Thl*  concept  1*  applicable  to  oost 
digital  control  environment* ,  depending  on  the  economic* 
of  the  application  regarding  fault  tolerance. 

4.5.  EXTENDABTLITT :  Extendabllity  proFably  do*«  not 
apply,  sine*  the  system  1*  still  loosely  specified. 

4.6.  CRITICALITIES:  The  system  is  no  t  cost-effective 
compered  to  other  eye t we  when  the  rwber  cf  faults  to  be 
tolerated  le  high  and  wh.-ra  ulf ro-’.ilgb  reliability  Is 
sought.  For  single-faulb  toloreoc*  end  less  high 
reliability,  the  eysten  configuration  right  be  changed. 

4.7.  IMPLICATIONS:  In  an  ultra-high  reliability 
application,  specification*  end  program  suet  be  proven  to 
be  correct.  In  this  system,  applications  programmers  suet 
alao  segment  thslr  pi, "-an*  Into  short  job  step*. 


5.  CONCLUSIONS 

5,1.1  STATUS:  Thla  Is  a  reaearch  project  with  a 
breadboard  experimental  unit  alnost  completed. 

5.2.  EXPERIENCE:  None  to  report  to  date. 

5.3.  FUTURE:  Some  pert*  of  the  system  still  need  to  be 
designed  end  prototyped.  Exeparlaents  must  be  conducted  on 
a  full-scale  prototype  system. 

5.4.  ADVANCES:  The  following  will  be  beneficial. 

‘Demonstrated  field  experience  with  varlow  fault-tolerant 
concept*. 

‘Practical  techniques  for  generating  correct  programs. 
‘Practical  wsye  of  verifying  that  a  program  1*  correct. 


6.  COMMENTS 

The  questionnaire  wee  good  in  the  sense  of  being  thorough, 
but  In  my  haste  to  respond  to  It  I  wonder  If  I  have 
omitted  significant  material.  An  additional  coament  about 
this  system  is  that  It  has  been  configured  around 
integrated  processors  end  memories  which  resemble  those 
that  era  available  today.  The  hardware  efficiency  nueber 
given  In  Section  4.3  le  very  misleading,  because  the  coat 
of  the  hardware  can  be  the  least  Important  cost  of  the 
system.  If  the  hardware  1*  conventional  end  not  overly 
expensive.  This  eye  tern  1*  expected  to  save  In  coats  of 
system  lntgegretlon,  program  verification,  and  operational 
reliability  experience.  These  savings  may  be  far  In 
excess  of  the  hardware  cost. 

As  an  additional  note,  the  replicated  approach  wad  here 
results  In  a  coverage  of  1.0  against  single  amlfwctlona. 
Coded  approaches  generally  give  lower  cov-iraga,  difficult 
to  quantify,  end  often  Impossible  to  verify  In  the  field. 


''  "Mi, 


P... Processor 
S... Scratchpad  memory 
M... Memory  module 
X... Multiplexor 
SSI. ..Subsystem  interface 


SURVEY  OF  FAULT-TOLERANT  COMPUTING  SYSTEMS 

W.  I  .  Martin,  Hughee  Aircraft  Company 
ullerton,  Celifornie  92634,  Mey,  1972 

1.  IDENTIFICATION 

1.3.  SUPPORT:  San*  aa  1.2 

1.4.  PARTICIPANTS:  The  participeting  organieationa 
incinde  NASA  MSFC,  Hughea  (eyetem  deeign),  M&S  Computing 

A.,buri*n*!:UtiV?  eoftw*re  under  eubcontrect  to  Hughai) ; 
Auburn  Univeraity  (executive  control  epproechea  imder 

TonZ?.  rrr  P«r«ciP»t.  by  ne«  "e  „ 
L  M^tin^Sc’  t'  c\Whlta*  Sherman  Jobei  Hughea  -  W. 

riwin  t4  '  M&S  *  T‘  T‘  Schen*m»ni  Auburn  -  Dr.  Devid 

hi',.  ?TjRT!  Th*  det*  of  conc*Ptl°n  «aa  clrce  1968  in  e 
concept  docwent  written  by  Dr.  White.  MSFC  haa  been 
developing  technology  under  their  Spece  Ultrarelieble 
Moduler  Computer  (SUMC)  program  eince  ehortly  thereefter 

Thi/ySt!im  da*ign  *ffort  being  performed  by  Hughea  waa 
-elicited  in  May,  1971  with  e  contrect  in  Oct^et" 

uiil'  C0MPLEyION-  The  Hughea  eyetem  definition  contract 
be  T”’  ln  APr11*  1S73.  Conatruction  of  e 
uncertain!  "  Pr°t0tyP*  may  foll«  “1th  completion  date 

1.7.  BIBLIOGRAPHY:  Verioua  planning  documente  have  teen 
"r‘““  NAEA'  Dr-  “b!"  m*y  be  contacted  for  theao. 
J*"Uhr  "f!ort  ,la  dlvld«d  Into  three  phaaea.  with  the 
Phaae  1  report  releaaed  on  April  13.  1972.  It  ie 
Deaign  of  a  Moduler  Digital  Computer  Syetem",  DRL  4) 

Phaae  1  Report,  Kughae  Aircraft  Company  FR  72-11-450.  Two 
other  papers  have  been  eubmittad  for  publication.  13,01^ 
fate  ie  uncertain  ae  of  yet,  but  intereated  pertiee  mey 

Conning:  ‘  °m  •  U  M,,rtln  “  Hugh“-  ">“«  e«  ^e 

BB1fRe5,,A  UntfloH  Method  for  Anelyeing  Mieeion 
Profile  Reliability  for  Standby  and  Multlpla  Moduler 
Redundant  Computing  Syetema  which  allows  for  Degreded 
Performance  (aubmltted  to  the  IEEE  Trane.ctione  on 
Reliability  Theory), 

*J.  L.  Bricker  and  W.  L.  Martin,  Reliability  of  Modular 
Computer  System,  with  Verying  Configuration  and  Loed 
Requirements  (submitted  to  1972  IEEE  Computer  Society 
Conference).  7 


2.  MOTIVATION 

2.1.  PURPOSE:  ARHMS  is  to  be  appliceble  through 
modularity  to  diverse  types  of  epece  miealona  ranging  from 
launch  vehicles,  to  apece  etetione  to  deep  apace  probea. 

2.2.  PHYSICAL  ENVIRONMENT:  Spacebome 

2.3.  COMPUTING  ENVIRONMENT:  See  2.1,  2.2. 

2.4.  COMPUTING  OBJECTIVES:  The  motivating  com -..ting 

objective  ie  to  ba  eble  to  configure  which  ere 

feult  tolerant  through  TMR  or  othar  redundant  modee  or  to 
use  tha  modulee  in  perallel  for  high  computing  cepeclty 
end  to  be  able  to  reconfigure  from  one  type  to  tha  other 
dynamically.  Maxlmta:  capacity  in  a  non-redundant  node  la 
to  be  eaverel  million"  additione  per  second. 


4.5.  RELIABILITY  OBJECTIVES:  One  specific  reliebilitv 

^  th“  the  Prob*bility  of  survival  of  »t“e..t 

fwlirir  CTPUt"  efter  5  ehould  be  at  least  0?99 

(with  no  on-board  maintenance).  The  overall  intent 
however,  is  thet  the  system  should  be  eble  to  be  * 

t0.m*et  »P«ific  mieaion  reliability  obje -lives 
whether  they  ba  steted  in  terma  of  maximum  recovery  time 
mxnber  of  failures  tolerated,  etc.  7  tlTO* 

2.6  DYNAMIC  VARIABILITY:  As  noted  in  2.4,  dynamic 
variebility  of  configuration  is  one  of  the  primery 
motivations.  K  ' 

2.?.  PENALTIES:  See  2.1, 

PHYSICAL  CONSTRAINTS:  There  are  no  explicit  physical 
constraints  axcept  those  implied  by  the  nature  of  the 
intended  apeceborne  application.  However,  an  implicit 
phyeical  constraint  ia  the  difficulty  of  contriving  an 
approach  to  a  large  (by  aeroapec.  atimderde)  computing 

walsh^A  fault-'°Jler«nt  design  within  the  confines  of 
weight  and  power  budgets  which  may  prevail  for 
interplanetary  mlsalona. 

2.9.  TRADEOFFS:  At  the  currant  stage  of  the  design  there 
are  many  critical  tradeoffs  yet  to  be  made. 

*  For  a  computer  which  will  be  built  after  1975.  what 
device  complexity  and  failure  rate,  aho.Ud  be  aaeianed? 

aTCt>  cf  th*  dealgn  ere  "itically  affected 
by  this  question.  Some  of  the  more  cruciel  ones  are  the 
maximum  complexity  of  any  module;  the  degree  to  which 
processors  must  he  sub-pertitioned;  the  reeulting  coat  in 
switching  hardware;  the  maximum  number  of  repllcetea  of 
any  one  module  type  which  must  be  eccommodeted;  and  th- 
complexity  of  tha  configuration  control  software. 

*  bf*^C  A*MMS  concept  developed  by  NASA  incorporates 
a  dedicated  executive  module  rather  than  e  floetina 
executive.  Resulting  tradeoffs  include  specific 

ef  nition  of  functions  to  be  performed,  specification  of 
atetus  monitoring  and  reconfiguration  parameters,  end  a 
deaign  approach  which  y  elds  sufficiently  high  reliability 
for  the  executive  module,  B  rsoniiy 

The  system  architecture  la  not  yet  defined  m  any 
complete  sense.  Questions  yet  to  be  resolved  include 
specific  definition  of  allowed  mode,  of  ope"! 
definition  of  the  maens  of  interconnecting  the  modules; 

f!rC^C  “a  U"e  Pf  VOt*rei  U8e  °f  error-correcting  codes 
for  memory  deta;  maximum  number  of  replicate,  per  module 

fiSlt’rT  techniques  for  memory  date  protection;  and 

fault  tolerence  feetures  within  each  module  cleee  At 
preeent,  we  are  making  tradeoffs  beaed  on  two  mejor 
configuration  alternatives.  Although  few  tradeoff 
conclusions  heve  been  reached,  the  predominating 
evaluation  criteria  are  almost  certain  to  be  the 
follcving: 

do.!°n^T!;a\IOn  fea8i,bllity  *  AnY  design  feature  which 
doe,  not  seem  to  ua  to  be  feasible  in  any  major  eenae 

5in  £ount>  ex-as8lv*  power,  deaign  coat)  will  be 
rejected.  We  are  not  particularly  Interested  in 
developing  new  theories  or  techniques  of  fault-tolerant 
computing  but  ara  very  interested  in  developing  a 

thTra"d5-dlo"I^ad.beaed  th*  "a“rCh  P«‘'~d  over 

*  Suitability  to  the  multi-mode  configuretion 
requirement,  -  ARMMS  ia  intended  to  be  uaable  in 
configurations  ranging  from  a  simplex  computer  to  TMR  wi  t. 
stendby  spares.  Any  feeture  which  imposes  excessive 
overheed  coat  for  the  banefit  of  one  configuration  at  the 
expense  of  others  is  suspect.  For  example”  edded  ha^are 
per  module  for  internal  fault  tolerance  multiplies  the 
hardware  penalty  paid  Jn  TMR  mode. 


3.  SYSTEM  DESCRIPTION:  As  scan  from  the  tradeoff 
discussion  above,  nc  firm  system  description  Is  possible 
now.  Therefore,  the  .esponsea  In  this  section  sre 
necessarily  brief  and  Incomplete. 

3.1.  ARCHITECTURE 

3.1.1.  CONFIGURATION 

3. 1.1.1.  INTERCONNECTIVITY:  All  processors,  I/O,  and 
executive  controller  may  access  all  of  main  memory  (a 
study  of  the  desirability  of  Identifying  an  additional 
level  of  memory,  cache  or  task  oriented  wae  made,  with  a 
negative  conclusion  reached).  The  most  probable  scheme  is 
a  system  of  replicated  busses  with  access  control  governed 
by  the  executive  module.  The  nature  of  spac^borne  I/O 

aeJfcVi^?  t*  bla?ln®  us  tow*rd  a  direct  processor  I/O  data 
path  which  can  be  used  for  transmitting  short  bursts  of 
dita.  The  executive  controller  will  monitcr  the  other 
system  modules  via  a  time-shared  bus.  This  bus  ordinarily 
polls  the  modules  in  sequence  but  may  be  interrupted  by 
the  processors  on  task  completion  or  other  time-critical 
event.  No  direct  interaction  of  moduleo  of  a  given  clasa 
(e.g.,  processor-to-processor)  Is  planned. 


3. 1.1. 2.  RANGE:  The  general  approach  to  achieving  the 
large  capacity  mentioned  previously  Is  to  maximize  the 
Individual  processor  performance  so  that  throughput  Is  not 
dependent  on  a  large  number  of  parallel  Instruction 
streams.  (Three  la  a  desirable  upper  limit.)  The  maxtmun 
main  memory  capacity  la  to  I  e  large  enough  (e.g., 

^3bK-512K  words)  to  support  the  high-throughput  goals. 

The  word  length  Is  to  be  32  bits  as  dictated  by  the  choice 
for  the  NASA  SUMC  processor.  Cumulative  I/O  data  rate 
capability  Is  to  be  10  million  bits  per  second.  In  all 
cases,  maximum  nunber  of  modules  per  class  (and  the  memory 
module  capacity)  will  he  determined  primarily  by 
reliability  considerations.  A  least  upper  bound  s  4  for 
each  class. 


3. 1.1. 3.  CAPABILITY :  (See  2.4.) 


4.4.  APPLICABILITY:  Applicability  to  other  than  space 
applications  Is  questionable. 

4.5.  EXTENDABILITY:  I  think  that  It  ie  more  likely  that 
the  system  design  can  usefully  contract  than  that  It  can 
be  usefully  extended. 

♦  CRITICALITIES;  The  major  difficulty  of  the  design 
la  the  breadth  of  the  goals.  The  critical  problem  Is 
therefore  to  find  a  set  of  design  choices  which  complies 
'.eaaonacly  well  with  all  the  goals  (e.g.,  we  want  high 
speed  and  capability  but  require  low  weight  and  power). 
However ,  I  don't  think  thut  slight  charges  would 
critically  affect  the  design.  (Also,  as  a  side 
observation,  while  one  Is  in  the  midst  of  a  system  design 
all  choices  seem  critical,  don't  they?) 

4.7.  IMPLICATIONS:  (Let  me  plead  that  this  question 
seema  too  vague.  I  don't  know  where  to  start  with  a  bilef 
response.) 


5.  CONCLUSIONS 

5.1.1  STATUS:  The  status  is  sufficiently  described  by  the 
above  comments,  I  think.  In  summation,  we  are  about 
one-third  of  the  way  through  a  system  definition  phase, 

5.2.  EXPERIFNCE:  It  appears  that  component  technology  la 
contributing  more  to  the  feasibility  of  highly  reliable 
machines  than  architecture  concepts  are.  As  recently  as  2 
or  3  yeara  ago,  gate  failure  race  of  10E-7  per  hour  seemed 
optimistic.  At  present,  gate  failure  rates  of  I0E-10  per 
hour  are  credible  for  the  space  environment.  On  the  other 
hand,  the  assumption  that  dormant  failure  rates  are  a 
small  fraction  of  active  failure  rates  appears 
questionable.  For  a  long-life  machine  In  an  unmanned 
environment,  these  two  factors  are  of  major  slgnificence 
to  the  system  designer. 


3.2.  FAULT  TOLERANCE:  (The  system  is  still  too  much 
conceptual  to  allow  a  decent  response.  All  faults  are  to 
be  tolerated.  None  are  to  be  not  tolerated.  All 
techniques  will  be  considered.  Ask  again  In  a  year  and 
let  s  see  how  It  turned  out.) 

3.3.  NOVELTY:  On  the  one  hand,  there's  nothing  that  one 
can  point  out  as  being  fimdamenti lx/  novel  (this  la  true 
of  most  machines,  I  think).  On  toe  other  hand,  there  are 
no  machines  that  1  know  of  that  have  successfully 
Implemented  a  variable  redundancy  approach  such  as  is 
being  sought.  The  choice  of  a  dedicated  executive  module 
is  the  only  deviation  at  the  block  diagram  level  from 
other  multiprocessors  (but  this  module  is  a  rather  close 
parallel  of  the  TARP  in  STAR). 


5.3.  FUTURE:  There  are  two  conflicting  poeelble  futures 
of  ARMMS.  The  pessimistic  view  la  that  It  will  go  the  way 
of  10  or  15  similar  paper  design  efforts  and  will  die  with 
only  e  final  report  to  commemorate  its  non-existence.  The 
optimistic  view  is  that  it  will  appeer  sufficiently 
promising  in  concept  that  NASA  wl l 1  continue  its 
development  and  eventually  attach  it  to  a  mission. 

Planning  Is  of  course  being  directed  toward  the  optimistic 
alternative. 

5.4.  ADVANCES:  I  cannot  add  anything  to  the  lists  of 
theoretical  problem  areas  and  needed  areas  of 
Investigation  which  SRI  described  in  its  reports  under 
contract  NAS  12-33,  In  particular,  I  agree  that  there 
have  been  too  few  case  studies  which  can  be  evaluated. 


3.4,  INFLUENCES:  JPL  STAR;  NASA  ERC  Modular  Computer* 
NASA  MSFC  SUMC;  IBM,  "Architectural  Study  for  a 
Self-Repairing  Computer;  SRI,  Techniques  for  the 
Realization  of  Ultra-Reliable  Spaceborne  Computers. 

3.5.  HARD-CORE:  The  executive  module  Is  hard-core.  The 
effect  Is  to  be  minimized  by  simplifying  the  module  as 
much  as  possible  end  by  internal  redundancy  (which  may 
ultimately  result  In  replication). 


4.  JUSTIFICATION 

4.1.  RELIABILITY  EVALUATION:  To  date,  reliability  haa 
been  evaluated  solely  by  analysis  (as  described  In  the  two 
papers  mentioned  in  1.7).  Later  in  the  effort,  we  expect 
to  extend  the  analysis  to  Include  coverage  and  switch 
unreliability.  We  also  expect  to  simulate  the  logical 
performance  of  the  Intermodule  switches  and  to  simulate 
the  injection  of  faults. 

4.2.  COMPLETENESS  OF  EVALUATION:  I'm  not  sure  that  I 
understand  the  question.  But  whatever  you  mean  by  design 
evaluation,  I'm  sure  that  I  wish  we  had  more  time  and 
money  to  do  it  better. 

4.3.  OVERHEAD:  Since  the  configuration  is  dynamic,  the 
percentages  of  resources  attributed  to  tha  achievement  of 
fault-tolerance  also  vary  with  time.  An  upper  limit  is 
probably  802;  a  lower  limit  Is  probably  202  (in  coat, 
logic,  execution  tima,  etc.). 


A  major  practical  advance  which  Is  needed  is  the 
Identification  and  exploitation  of  specific  spplications 
in  which  fault-tolerant  machines  can  be  Justified 
economically.  It  Is  significant,  I  think,  that  the  Bell 
ESS-I  and  System-3th  FLT's  instruction  retry,  etc., 
represent  the  most  e*.  ensive  application  of 
fault-tolerance  and  diagnostic  techniques.  Both  sre  In 
areas  where  the  payoff  for  high  reliability  is  great. 
Although  aerospace  applications  have  supported  much  of  the 
research  in  fault-tolerant  machines,  I  am  skeptical  that 
there  is  a  sufficient  mass  of  money  there  to  lead  to  very 
widespread  results  ir,  fielded  systems.  The  situation  is 
analogous  to  that  wnich  has  existed  for  associative 
processing  fer  10  years,  in  that  the  glamour,  concepts, 
and  techniques  are  often  apparent  but  cost  considerations 
ultimately  lead  to  more  conventional  choices. 

Also,  I  wonder  if  "fault-tolerant  computing"  Is  too  narrow 
a  view  and  that  many  of  the  basic  ideas  would  be 
applicable  to  a  discipline  of  "Fault-tolerant  systems". 
Terhaps  there  are  other  equally  fertile,  but  less  plowed 
fields  to  be  conquered. 


6.  COMMENTS:  (See  5.4) 


SURVEY  OF  FAULT-TOLERANT  COMPUTING  SYSTEMS 

John  H.  Vans lay,  Stanford  Reeaerch  Institute 
Menlo  Park,  Cs.  94Q25,  May  1972 

1.  IDENTIFICATION 

1,1,  NAME:  SIFT  (Software-Implemented  Feult  Tolerance), 
project:  daslgn  study  of  a  fault  tolerant  digital 
computer 

3.2  RESPONSIBILITY:  SRI 

1.3  SUPPORT:  NASA  Langley 

1.4.  PARTICIPANTS:  J.  Goldbarg,  K.  Levitt,  R.  Ratner,  J. 
Wens  ley,  H,  Zeidler,  M.  Green 

1.5.  START:  August  1971 

1.6.  COMPLETION:  Experimental  vareion  1973,  final  design 
1974 

1.7.  BIBLIOGRAPHY:  Technical  Prograaa  Narratives  1-7; 
"SIFT  -  Software  Implemented  Feult  Tolarance,”  submitted 
to  FJCC  1972 


2.  MOTIVATION 

2.1.  PURPOSE:  Control  processing  in  an  advanced 
technology  traneport  (aircraft)  including  navigation, 
stability  augmentetlon,  angina  control,  instrument  blind 
landings,  etc. 

2.2.  PHYSICAL  ENVIRONMENT:  Airborne  —  the  system  concept 
howaver  ia  eppllcable  to  any  environment. 

2.3.  COMPUTING  ENVIRONMENT:  Real-time 

2.4.  COMPUTING  OBJECTIVES:  Configuration  scaleability , 
graceful  degradation,  traneportebility  of  concept  to  any 
procasaor  or  memory  design. 

2.5.  RELIABILITY  OBJECTIVES:  Minimum  probability  of 
arronaoua  results,  and  of  loss  of  com,  iting  capacity 
during  aircraft  flight. 

2.6.  DYNAMIC  VARIABILITY:  Variable  degrees  of  feult 
tolerance  for  tasks  of  differing  criticality.  Ability  to 
trade  off  batvaen  computing  power  and  fault  tolerance. 

2.7.  PENALTIES:  Worst  case  -  human  lives;  intermediate  - 
aircreft  damage;  least  case  -  need  to  ebort  flight 
objectives. 

2.8.  CONSTRAINTS:  h^rdwera  must  be  daeignad  with  weight, 
else  and  power  requirements  consistent  with  aircraft 
requirements,  Th«*  basic  concept  of  the  system  ie  only 
effected  by  tha  constraint  that  maintenece  cannot  be 
cerriad  out  during  flight, 

2.9.  TRADEOFFS:  Computing  capacity  vs.  reliability 

3.  DESCRIPTION:  A  system  erchitecture  in  which  fault 
tolerance  is  achieved  with  no  special  fault- tola rant 
hardware, 

3.1.  ARCHITECTURE:  A  multi- computer  (see  Fig  1) 

3.1.1.  CONFIGURATIONS:  No  constraints  are  present  on 
processor  or  memory  daslgn.  Fault  tolerance  is  achieved 
by  tha  restricted  connection  of  processors  and  memories, 
and  by  software  control. 

3. 1.1.1.  INTERCONNECTIVITY:  Processing  modules  comprising 
e  procaseor  and  memory  are  connected  via  multiple  buases. 
Tha  interconnection  is  designed  so  that  processors  may 
only  read  (and  not  write)  into  the  memory  of  other 
modules.  The  buases  are  used  as  alternative  routes  rathsr 
than  as  multiple  simultaneous  tranemlasion  paths. 

3. 1.1.2.  RANGE:  Tha  scale  of  the  system  is  not  froien  in 
the  architectural  concept.  It  is  envisaged  that  a  minimum 
cou figuration  would  contain  three  processing  modules  and 
three  busses.  The  design  does  not  (at  present)  place  any 
limit  on  the  maximum  configuration.  Greater  feult 
tolerance  is  achieved  with  a  larga  number  of  low- 
capability  units  rather  than  with  a  small  number  of  high 
capability  units. 


3. 1.1. 3.  CAPABILITY:  Tha  design  concept  is  valid  over 
the  entire  range  of  processor,  memory  and  bus  capability. 

3.1,2.  EXECUTIVE:  Executive  control  (allocation, 
scheduling,  dlapetchiug,  reconfiguration,  ate.)  is 
achieved  by  raplicatad  software  executive  routines. 

3. 1.2.1.  MODES:  Tha  primary  operating  mods  is  on 
repetitive  raal-tlme  calculations  involving  many  loosely 
connected  tasks.  Both  multiprocessing  and 
multiprogramming  are  included. 

3. 1.2.2.  SOFTWARE:  Teska  ara  multiprogramned  in  each 
processing  module.  Each  task  for  which  fault  tolerance  is 
demanded  le  present  in  more  than  one  module.  A  loose 
synchronization  of  t^ek  processing  is  achieved  by  the 
system  executive  (which  itsalf  is  rsplicated  and  loosaly 
synchronized).  Software  fault  detaction  ia  csrriad  out 
between  each  iteration  of  a  task  before  erroneous  results 
era  used  by  the  next  its  ration  or  other  tasks. 


Figure  1  System  Con f  igu ra  l. ion 


3.2  FAULT  TOLERANCE 

3.2.1.  FAULTS  TOLERATED:  The  syatem  la  tolerant  to  faults 
in  any  unit  (proceasor,  bus  or  memory).  The  faults  may  be 
the  erroneous  result  of  an  action  (calculation, 
transmiaaion  or  storage)  or  the  failure  of  a  unit,  to  carry 
out  any  action. 

The  syatem  handles  transient,  and  permanent  faults, 
treating  long-term  intermittent  faults  as  permanent.  The 
reconfiguration  procedurea  can  bring  back  Into  service  s 
unit  tb at  was  at  one  time  subject  to  faults  but  has  since 
recovered  or  been  repaired. 

The  cause  of  the.  fault  (electrical,  mechanical,  etc.)  Is 
not  of  importance,  the  only  consideration  is  whether  the 
results  of  actions  In  replicated  units  agree  or  disagree. 

independent  multiple  faults  can  be  tolerated  to  any  degree 
depending  on  the  extent  of  replication  of  the  function. 
Correlated  faulta  both  in  hardware  and  software  aro  not 
tolerated  to  the  same  extent  as  uncorrelated  faults.  The 
loose  synchronization  of  tasks  assists  in  tolerating 
faults  which  are  correlated  in  time  rather  than  function. 
One-shot  faulta  do  not  cause  removal  or  reconfiguration  of 
units  from  the  system.  The  propagation  of  a  fault  from 
any  unit  to  another  can  only  occur  if  both  units  are 
faulty. 

3.2.2.  FAULTS  NOT  TOLERATED:  Multiple  correlated  faults 
that  are  not  detected  by  a  voting  procedure,  or  by 
repeating  the  task,  e.g. ,  simultaneous  identical  failure 
of  two  memory  units  when  threefold  replication  is  used. 
Masaive  faults  that  reduce  the  system  to  a  size  too  small 
to  handle  the  computing  load. 

3.2.3.  TECHNIQUES;  Fault  detection  is  carried  out  by 
replication  and  voting.  Other  fault  detection  methods 
(hardware  or  software)  are  compatible  with  and  can  be 
incorporated  into  the  aystem  concept.  Fault  correction 
(or  tolerance)  is  achieved  by  voting  after  replication  in 
most  cases  but  can  be  supplemented  by  other  techniques 
such  as  repetition  or  roll-back.  The  allocation  of 
resourcea  to  tasks  can  be  changed  either  when  faulty  units 
are  removed  or  when  the  mission  demfjida  different  fault 
tolerance  and/or  computational  power. 

3.3.  NOVELTY:  Lack  of  need  for  special  hardware  units  to 
facilitate  fault  tolerance.  Ability  to  t -ade  off  fault 
tolerance  with  computing  power.  Applicab. lity  of  the 
system  concept  to  different  memory  or  processor  designs. 


3.4,  INFLUENCES:  The  design  is  Influenced  by  the  need  to 
avoid  special  hardware  for  fault  tolerance,  freezing 
fault  tolerance  techniques  at  design  time,  designs  geared 
to  particular  aize  and  speed  computers. 

3.5.  HARD  CORF:  i  don't  mean  anything  by  "hard  core"  in 
the  system  described.  I  can  imagine  other  system  concepts 
in  which  the  term  has  meaning  (but  little  utility). 

4.  JUSTIFICATION 

4. 1.  RELIABILITY  EVALUATION:  By  analysis,  assuming 
incorrelated  faults  of  equal  probability  in  each  part  of 
the  system  (chip,  connector,  cable,  etc.). 

4.2.  COMPLETENESS  OF  EVALUATION:  Incomplete. 

4.3.  OVERHEAD:  Variable,  typically  a  3-i  cost  pensity  Is 
paid  for  .‘fault  tolerance. 

4.4.  APPLICABILITY:  General;  the  design  is  applicable  to 
any  environment. 

4.5.  EXTEN DAB i LITY:  Unlimited. 

4.6.  CRITICALITY:  Multiprocessing  is  critical. 
Multiprogramming  Is  highly  desirable  (see  Fig  2). 

4.7.  IMPLICATIONS:  There  are  no  implications  on  the 
hardware  designers  of  processors  and  memories.  The  busses 
are  constrained  in  the  way  units  communicate.  The 
applications'  software  must  be  implemented  so  that  input 
data  for  a  program  is  fetched  by  calling  a  general  system 
routine  which  carries  out  fault  detection  and  correction. 


5.  CONCLUSIONS 

5.1.  STATUS;  A  conceptual  design  of  hardware,  software 
and  fault  tolerance  procedures  exists. 

5.2.  EXPERIENCE:  Software  design  studies  show  that  the 
time  and  memory  requirements  of  the  fault  detection  and 
correction  routines  are  reasonable. 

5.3.  FUTURE:  The  projection  is  for  an  experimental 
version  of  the  aystem  to  be  built. 

5.4.  ADVANCES:  I/O  units  with  fault  tolerance 
capability. 
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SU»m  07  PAULI-TOLERANT  CONFUTING  SYSTEMS 

Barry  X.  Borgaraon,  Computer  Syataaa  Research  Project 
University  of  California,  Berkeley,  May  1972. 


i.  identification 

1.1.  NAME!  PRIME 

1.2.  XESP0NSIB1LITT I  Conputar  Systems  Research  Project 
(CSRP),  U.  C.  Berkeley 

1.3.  SUPPORT!  AEPA  -  Contract  No.  DAHC70  15  C  0724 

1.4.  PARTICIPANTS!  Rtrbort  1.  Baaklo,  Principal 
Investigator!  Roger  Roberts,  Principal  Progmaari  lorry 
R.  Borgaraon,  Hied,  Hardware  RAD. 

1.5.  START!  7/1/70 

1.4.  COMP  LI  Tick’!  '  prototype  to  ba  naming  about  7/73 
1.7.  B1BL10CU. 
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•Borgaraon,  Barry  R. ,  "A  Pall-Softly  Syatan  for  Tina 
Sharing  Uaa,"  Dlgaat  of  f^n  1972  Intarnatlonal  Fault 
Tolarant  Conputlng  Syipoolun. 

•Quatse,  Jaaaa  T. ,  Plarra  Caul ana  and  Donald  Dodga,  "Tha 
Extamal  Aceaaa  Network  of  a  Modular  Conputar  Syatan," 
Procaadlnga  of  tba  1972  SJCC,  AFIPS  Praaa,  pp.  713-790. 

•Fabry,  F.  S.,  "Dynanlc  Varlflcatlon  of  Operating  Syatan 
Dacia loon ,”  CSRP  Dociawnt  No.  P-14.0,  Unly.  California, 
Parka lay,  Iaauod  2/23/72. 

•Borgaraoo,  Barry  X, ,  "Spontaneous  Raconflgvr.tlo*  In  a 
Pall-Softly  Conputar  Utility,"  CSRP  Docuaeat  No.  P-15.0, 
Unly.  California,  Xarkalay,  laauad  2/29/72. 

•Borgaraon,  Barry  X.,  "Dynanlc  Confimatlon  of  Syatc 
Integrity,"  CSRP  Docwnet  No.  P-19.0,  Unly.  California, 
Barkalay,  laauad  4/24/71. 


2.  MOTIVATION 

2.1.  PURPOSE!  Cenarel-purpoaa,  lo tnrnct Ira,  nultl-accaaa 
conputlng. 

2.2.  PHTS1CAL  ENVIRONMENT !  Ground  baaad 

2.3.  COMPUTING  ENVIRONMENT!  Ranota  accaaa  oyar  talaphona 
lloaa  and  ayantually  oyar  tha  Arpanet. 

2.4.  COMPUTING  OBJECTIVES!  Thin  la  not  tha  primary 
motivation  area  lo  our  ayatan  daalgn.  Ue  anticipate  that 
tha  orlgloal  cooflguratlon  of  PRIME  will  a up port  about  100 
uaara  with  a  wo  rat  caaa  raaponaa  tlna  of  laaa  than  two 
aaconda  for  trivial  Jobs. 

2.5.  RELIABIL1TT  OBJECTIVES!  Sacauaa  wa  will  ba  able  to 
rapalr  unite  aa  they  btcona  faulty,  wa  are  mining  for 
coutlnuoua  availability.  Tha  ayatan  parfornnaca  ahould 
oavar  degrade  below  75Z  of  lta  peak  capacity. 

2.6.  DTNAM1C  VARIABUTTi  Parforuanca  cannot  ba 
dynanlcally  traded  for  reliability.  However,  provlalona 
nay  aonaday  ba  added  which  will  allow  dynamically  trading 
performance  for  lotraprocaaa  Integrity  (Saa  lection  6). 

2.7.  PENALTIES!  Tha  effecte  of  lntraprocaea  data 
contamination  (Saa  Section  3.3.2)  due  to  ayatan  failures 
will  atrougly  depend  on  tha  nature  and  purpnaa  of  tha 
proeaaa.  There  aaanu  to  ba  no  way  to  ganarnllaa  about 
thla.  If  tha  ayatan  ltaalf  ware  to  rraah,  thin  would  oo 
doubt  load  to  a  loaa  of  revenue  If  PRIME  ware  tranaferrad 
to  a  counarclal  anvlroanant, 

2.8.  CONSTRAINTS!  There  are  oo  apaelflc  conatralata  of 
alaa,  weight,  and  power.  Tha  aalf-lnpoaad  conatralnt  on 
coat  la  to  try  to  build  a  fault-tolerant  ayatan  that  la  aa 
done  In  coat  aa  poaelbla  to  any  currant  ayatan  with 
comparable  power  tad  cepabllltlea , 

2.9.  TRADEOFFS!  (Too  complicated  to  deal  with  briefly! 
ana  Sectlona  4,4,  4.6  and  6.) 


3.  DESCRIPTION 

3.1.  ARCHITECTURE 

3.1.1.  CONFIGURATION 

3. 1.1.1.  INTERCOM rtCTlVTi'?!  Figure  1  la  a  block  diagram  of 
PRIME.  Tha  External  'cease  Network  (BAN)  allova  any  pro- 
c earner  to  connect  to  any  dlak  drive,  external  device,  or 
other  proeaaaor.  Each  pro caaa or  haa  three  eueh  lndepeudeot 
petha  Into  tha  EAM.  Tha  EAN  connectivity  ranalna  univer¬ 
sal  over  tha  different  ayatan  sltas.  Unlvaraal  awltchlog 
between  all  procaaaora  and  all  nanory  blocks  la  not  provi¬ 
ded.  Instant,  each  proeaaaor  always  connects  to  exactly 
64K  of  maaory  ragardlaaa  of  tha  alaa  of  tha  ayatan. 

3.1. 1.2.  RANGE!  Tha  PRIME  archltactura  will  usefully 
eceonodata  from  3  to  about  30  procaaaora.  Each  processor 
could  connect  to  from  16K  to  128R  of  primary  memory. 
Depandlog  on  tha  type  of  disk  drlvaa  used,  from  1  to  5 
drives  par  processor  would  ba  reasonable.  Tha  currant 
ayataa  has  bean  designed  to  operate  with  from  tbraa  to 
tight  procaaaora  without  requiring  mj  additional  hertbtara 
or  software  daalgn.  Useful  nanory  alaas  range  from  64K  to 
about  256E.  Disk  drlvaa  range  from  about  alx  to  24.  Each 
proeaaaor  to  ba  used  lo  tha  Initial  Imp lamentation  of 
PRIME  will  ba  a  Mata  4  (Digital  Seiaotlfle  Corp.).  Tha 
Mata  4  la  a  ganaral-purpoas,  16-bit,  32-raglatar,  9 One- 
eye  la  tlna  microprocessor.  Tha  nanory  la  33  bits  wide, 
about  600  ns  cycle,  and  nada  from  1024-blt  NOS  chips.  Tha 
dlak  drlvaa  art  double  (track)  density  2314-typa  drlvaa 
that  hava  bean  modified  to  transfer  lafoiaatloa  on  two 
haada  at  a  tins.  Tha  Initial  configuration  will  hava  five 
procaaaora,  104K  of  nanory,  and  IS  disk  drlvaa. 

3.1. 1.3.  CAPABILITTi  Tha  capability  la  not  accurately 
known  at  thla  tlna. 

3.1.2.  EXECUTIVE 

3. 1.2.1.  MODES!  At  any  given  tins,  oaa  proeaaaor  la 
designated  tba  Cootrol  Processor  (CP)  while  tha  rest 
function  aa  Problem  Procaaaora  (PPa).  Uaar  procaaaaa  arm 
na  on  tha  PPa,  Multiprogramming  la  not  used,  but 
procaaaaa  are  overlap-swapped.  In  order  to  achieve  a  vary 
high  Interprocess  lotegrlty.  It  was  decided  never  to  1st 
two  procaaaaa  share  nanory!  hence,  coeparatlva-procaaa 
multiprocessing  la  not  poaalble  with  PRIME. 
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3. 1.2. 2.  SOFTWARE!  The  system  software  la  divided  Into 
thraa  aactlona.  Thara  la  tha  Cantral  Control  Monitor 
(CCM)  which  runs  on  tha  Targat  Maehloa  of  tha  CP;  tha 
Extanslon  of  tha  Control  Monitor  (ECU)  which  raaldaa 
directly  In  tha  nlcrocoda  of  aach  procaaaor;  and  tha  local 
Monitor  (LM)  which  r taa  on  tha  Targat  Maehloa  In  tha  PPa. 
Tha  CO  la  raaponalbla  for  achadullng  procaaaaa,  allocat¬ 
ing  raaourca,  and  consumutlng  lntarprocaaa  aaaaaga 
tranafara.  Tha  ECU  lncludaa  tha  disk,  tarnlnal,  and 
coaaiunlcatlon  controllers,  logic  for  double-checking 
critical  CCM  decisions,  bootatrap  logic,  and  aoaa  lntalll- 
ganca  to  daal  with  ra configuration,  Tha  LM  contalna  tha 
flla  and  vorklng-eet  management  systems.  Tha  CO  doaa  not 
gat  involved  with  a  procaaa  aftar  It  haa  started  tha 
procaaa  up,  Tha  procadura  followad  by  tha  CO  la  to 
alloeata  tha  nacaaaary  raaourcaa,  lnltlata  tha  roll  In, 
and  lat  tha  LM  aod  EO  taka  over  from  thara.  Tha  CO  will 
not  gat  Involved  again  until  tha  procaaa  althar  tlwa  out 
or  blocka  ltaalf,  Tha  LM  daala  only  with  uaar  procaaaaa; 
It  la  completely  laolatad  from  tha  raat  of  tha  ay a taa. 
Because  of  thla,  uaara  will  ba  fra.  „„  provlda  thalr  own 
LM  If  thay  do  not  Ilka  tha  atandard  ona  provided. 

3.2.  FAULT  TOLERANCE 

3.2.1.  FAULTS  TOLERATED;  PRIME  will  tolarata  all 
lntarnal  fault..  That  la,  tha  ayatan  la  azpactad  to 
continue  oparatlng  even  In  tha  praaance  of  any  arbitrary 
aoftwara  or  hardwara  faulta.  Tha  ayatan  will  raconflgura 
to  run  without  any  placa  of  hardwara  that  becomes  faulty, 
and  aiaehanlana  azlat  for  llalf<-g  tha  affacta  of  any 
aoftwara  fault.  PRIME  haa  bat.,  daalgnad  t<  prowl  da 
continue ua  aarrlca  to  (almost)  all  terminals.  In  Boat 
caaaa,  a  faulty  unit  will  ba  rapalrad  and  ratumad  tt 
aarvlca  bafora  anothar  failure  occura.  However,  tha 
eyataai  will  atlll  continue  to  operate  with  a  auhatantlal 
part  of  tha  raaourcaa  reaoved  from  actlva  uaa ,  Tha  ayatan 
ahould  alnoat  navar  degrade  to  below  73  percent  of  lta 
naxlmaa  capacity.  In  addition  to  continuity  of  aona 
mlnlmta  aarvlca,  lntarprocaaa  Integrity  vlolatlona  are 
prevented  at  all  tliMa;  thla  lncludaa  tha  relatively 
unatable  parloda  between  tha  oneet  of  a  fault  and  tha 
detection  and  leolatlon  of  tha  faulty  unit. 

3.2.2.  PAULTS  NOT  TOLERATED:  Only  tnvlronnantal  faulta 
are  not  tolaratad  by  PRIME.  Tha  moat  common  of  thaaa 
faulta  would  ba  In  tha  A.C.  power  and  air  conditioning. 
Slnca  It  la  aaay  to  aaa  hew  to  back  thaaa  raaourcaa  up,  no 
effort  haa  bean  made  to  Incorporate  fault  tolaranca  with 
raapact  to  thaaa  unlta  wlthlln  PRIME,  uhlla  PRIME  aa  a 
ayatan  will  continue  to  run  In  aplta  of  lntarnal  fallurea, 
Individual  procaaaaa  may  occaalonally  gat  clobbered.  That 
la,  no  apedal  provlalona  haw  bean  eada  In  PRIME  t< 
guarantee  lntarprocaaa  Integrity.  Hence  tranalent 
fallurea  will  frequently  cauea  contamination  of 
Intonation  for  aona  procaaa.  Alao  hard  falluraa  will 
often  clobber  ona  procaaa  bafora  being  detected.  Tha  .no at 
aerloua  dleruptlon  will  probably  occur  whan  a  dlak  drive 
falle.  Whan  thla  happen. .  all  of  tha  procaaaora  that  ware 
ualog  that  drive  will  ba  auapendad  until  an  operator  can 
recover  thalr  data,  althar  by  moving  tha  dlak  pack  to 
anothar  drive,  or  recovering  from  tapaa  In  tha  unlikely 
event  of  a  head  craah.  But  avan  In  thla  vorat-caae 
cataatrophy,  only  a  email  part  of  tha  uaara  (about  7 
percent  In  tha  Initial  ayatan)  will  ba  affected. 

3.2.3.  TECHNIQUES;  Tha  baalc  ayetem-vlda  technique  uaad 
to  achieve  fault  tolaranca  la  to  allow  the  ayatan  to 
degrade  gracefully  by  reconfiguring  to  run  without  any 
favlty  unlta.  At  the  heart  of  tha  achame  la  a  distributed 
architecture  with  a  multiplicity  of  all  functional  unlta 
except  tha  EAR,  which  la  daalgnad  to  fall  eoltly  on  lta 
own.  Fault  detection  la  acconpllahad  by  a  variety  of 
method,  which  Include  parity  menory  and  bueea,  survelll- 
anca  taata  on  aach  procaaaor  aftar  aach  Job  atap,  a  double 
check  on  all  critical  ayatan-wlda  daclalona  made  by  tha 
CP,  and  fault  Injection  In  auch  area,  aa  error  datactorn 
and  tha  aaldon  uaad  reconfiguration  logic.  Aftar  a  fault 
la  detected,  an  Initial  reconfiguration  ceuaaa  a  procaaaor 
not  lovolvad  In  tha  detection  to  become  tha  new  CP.  Thla 
virtual  "hard-core"  than  loltlataa  dlagnoatlca  to  locate 
tha  faulty  unit,  laolata  It,  and  raconflgura  tha  system 
to  run  aa  efficiently  aa  poealbla  without  It.  A  email 
amount  of  dedicated  hardware  aaeoclatad  with  aach 
procaaaor  guarantaaa  that  tha  Initial  reconfiguration  will 
ba  accompllehad  properly.  It  la  poealbla  to  logically 
laolata  aach  major  \alt  at  lta  eyetem  boundarlaa  ao  that 
tha  eyetem  can  nn  flna-eaah  dlagnoatlca  or  exerdae  tha 
hardwara  to  aid  In  locating  tha  faulty  component.  In  tha 
caaa  of  a  failure  of  tha  leolatlon  logic,  any  unit  can  ba 
dynamically  powered  down  to  provlda  guaranteed  laoletlon 
from  tha  raat  of  tha  ayataa. 


3.3.  NOVELTY;  Tha  dlatrlbutad  nature  of  tha  ayatam. 
Including  tha  dlatrlbutad  lntalll ganca  In  tha  form  of  the 
ECMa,  provide,  a  v*r  owtirful  atructura  whereby  fault 
tolaranca  la  achieved  without  tha  uaa  of  any  "reliable" 
hardwara.  Vary  high-performance  low-coat  dlak  drlvaa  have 
bean  Incorporated  In  auch  a  way  aa  to  allow  thaaa  devices 
to  be  used  as  second  level  storage,  third  level  storage, 
and  tha  swapping  medium.  By  distributing  these  three 
functions  over  many  Identical  physical  units,  vary  high 
availability  Is  achieved  at  whet  la  actually  a  lowar  coat 
and  with  hlghar  overall  performance  than  would  ba  possible 
with  thraa  dlatlnct  type,  of  unlta.  PRIME  automatically 
raaponda  to  faults  by  reconfiguring  to  run  without  tha 
faulty  unit.  Since  thara  la  a  multiplicity  of  all 
functional  unlta  except  tha  EAN,  It  la  quits  aaay  to  rtai 
without  any  particular  unit.  Rather  than  make  tha  EAN 
"reliable,"  a  more  economical  approach  waa  taken  whereby 
carefully  controlled  failure  modes  wars  daalgnad  Into  It. 
Thla  results  In  a  failure  within  tha  EAN  manifesting 
Itself  aa  a  failure  of  a  small  ntaber  of  ports,  which  la 
equivalent  to  losing  whatever  la  attached  to  those  porta, 
and  the  ayatam  waa  already  daalgnad  to  handle  that 
eventuality.  Tha  reconfiguration  structure  la  alao  vary 
Interesting.  Whenever  a  failure  la  detected,  an  Initial 
reconfiguration  takas  placa  which  establishes  a  new 
processor  as  tha  CP.  Tha  new  CP,  which  la  ona  not 
Involved  In  tha  detection  of  tha  fault,  Is  than  uaad  as 
tha  temporary  "herd-core"  to  lnltlata  dlagnoatlca,  locate 
tha  fault  If  Indeed  it  exists,  and  remove  tha  faulty  unit 
from  tha  ayatam.  Tha  dlatrlbutad  intelligence  of  PRIME 
has  been  uaad  to  provlda  double  checking  on  all  critical 
ayatam  functions,  which  In  turn  guarantees  that  thara  will 
ba  no  lntarprocaaa  lntarfaranca.  Probably  tha  most 
unusual  general  feature  of  PRIME  with  raapact  to  fault 
tolaranca  la  that  It  la  self-dlagnoslng  and  salf-rapalrlng 
without  Incorporating  any  "hard-core." 

3. A.  INFLUENCES:  Many  previous  efforts  hava,  of  course, 
Influenced  us,  but  no  slngla  system  stands  out  aa  having 
special  Influence. 

3.5.  HARD-CORE;  No,  thara  la  no  "hard-core”  In  PRIME. 
Instead,  tha  concept  of  a  "floating  hard-cora"  exists 
wharaby  a  working  procaaaor  la  pressed  Into  aarvlca  aa  tha 
Control  Procaaaor  whenever  a  malfixictlun  la  datactad. 

This  la  coma  latent  with  tha  overall  system  phlloaophy  of 
not  having  any  "reliable"  hardware  anywhere  In  tha  system. 


A.  JUSTIFICATION 

A.l.  RELIABILITY  EVALUATION;  Reliability  will  ba 
demons tratad  by  stimulation  of  faults. 

A. 3.  OVERHEAD;  Tha  coat  of  tha  additional  hardware  that 
has  bean  Incorporated  in  PRIME  apedflcslly  for  fault 
tolaranca  la  laaa  than  10  percent  of  tha  total  hardware 
coat  of  tha  ayatam,  Laaa  than  10Z  of  each  processor's 
useful  time  la  davoted  to  fault-' parent  functions,  slnca 
tha  surveillance  programs  era  run  during  what  would 
otherwise  ba  Idle  time  while  procaaaaa  are  being  avappad. 

A, A.  APPLICABILITY:  PRIME  haa  bean  vary  carefully 
designed  to  perform  economically  In  a  particular  anvlron- 
ent.  If  It  waa  to  ba  uaad  In  anothar  environment,  a 
detailed  analysis  would  hava  to  ba  performed  to  determine 
what  changes  would  hava  to  ba  made  to  allow  It  to  perform 
adequately  In  the  new  environment .  In  particular,  moat 
other  potential  environments  would  require  that  atapa  ba 
taken  to  guarantee  lntraprocasa  Integrity. 

A, 6,  CRITICALITIES:  Tha  choice  of  dlak  drlvaa  la  quite 
critical  since  a  low  coat/blt  la  oacaaaary  aa  wall  aa  a 
high  bandwidth  dua  to  the  different  functions  thaaa  drives 
perform.  Slnca  3330-typa  drives  wars  not  available  whan 
thla  design  started,  2314-typa  drlvaa  ware  selected  and 
modified  to  transfer  at  SMHt.  Alao,  tha  EAN  had  to  ba 
carefully  designed  with  wall-apaclflad  failure  modes. 
However,  tha  primary  memory  and  tha  procaaaora  are  simply 
"off  tha  shelf"  ltaete.  As  for  goals,  tha  decision  to  not 
provide  lntraprocasa  Integrity  checks  has  basn  carefully 
exploited  In  tha  design  of  PRIME  and  has  provided  a  vary 
substantial  cost  savings. 
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4.7.  IMmCAIlOMIt  Heavy  rellaeea  la  placad  on  parlodle 
cheeking  of  hardware  rathar  than  eooeurrant  checking. 

Thus,  tha  ebllity  to  Inject  faults  Into  tha  approprlata 
araaa  haa  baan  a  difficult  requirement  placad  on  all  of 
tha  hardware  daalfnara.  Tha  moa t  notable  software 
raquliaaaot  luptaad  by  tha  baalc  design  la  tha  dear 
division  of  tba  operating  ayatea  Into  three  parte,  one  of 
which  can  ba  furniabad  by  a  uaar.  Tha  only  significant 
requirement  placad  on  a  uaar  la  that  ha  auet  ba  awa r»  that 
no  lntraprocaaa  Integrity  chac:  a  era  -e'V*  (Just  Ilka  In 
all  currant  tlaa-aharlng  eya<eae) 


5.  CONCLUSIONS 

5.1.  STATUSi  Tha  deelgn  of  P7IMR  la  about  95  percent 
completed,  and  laplaaantatlor  haa  baguu  on  both  tha 
hardware  and  aoftwara.  Tha  ftrat  vara  Ion  capable  of 
reconfiguring  In  tha  praaant  of  a  failure  ahould  ba 
running  by  Auguet,  1973. 

5.2.  EXPERIENCE t  Tha  naln  conclualon  that  tha  raepondant 
can  uaka  regarding  tba  daalgn  of  PRIME  la  that  by  aoaawhat 
Halting  tha  goal  of  tha  PRIME  system,  It  waa  poaalbla  to 
create  a  eyatan  that  ahould  exhibit  excellent  fault- 
tolerant  charactarlatlca  at  a  much  lower  lncraaantal  coat 
than  tbat  of  any  other  feult-tolarait  ayatan  known  to  hln. 

5.3.  FUTURE i  Tha  near  future  will  ba  davotad  to  building 
PRIME.  After  tbata  evaluation  and  twlng  will  taka  place 
with  connection  to  tha  Arpanet  wary  likely. 

5.4.  ADVANCES t  It  aaaaa  that  tha  eoat  algnlfleant 
development  that  would  aid  tha  PRIME  system  would  ba  tha 
availability  of  a  general-purpose,  aal f-cha eking 
procaaaor.  Since  100  pereaet  ealf-chackablllty  la 
axtraaraly  difficult  to  daalgn  Into  a  procaaaor,  tha  beat 
couraa  of  action  hara  aaaaa  to  ba  to  wait  for  LSI 
procaeeore  of  eufflclant  power  to  ba  built.  Thaaa 
proeaaanra  ahould  ba  ao  lnexpanalva,  eoaparad  to  tba  raat 
of  tha  hardware  coat,  that  running  two  of  than 
elnultanaouely  and  ccaparing  outputa  ahould  ba  a  vary 
attractive  procedure  economically.  In  fact,  tba  current 
procaaalng  element  In  PRIME  could  ba  broken  Into  aavaral 
aubprocaaaore :  one  for  coaaiBl  cat  Iona,  one  for  tha  dlak 
controller,  ona  for  tha  taralnal  controller,  two  for  tha 
Target  Machine,  ate.  Probably  only  tha  Targat  Machine 
procaaaor  would  have  to  ba  dvylaxad  bacauaa  tha  o there  can 
have  Independent  chacka  on  tha  validity  of  thalr  reaulta. 
With  thla  procedure,  lntraprocaaa  Integrity  would  ba 
poaalbla  at  an  lnalgnlflcant  lncraaantal  coat. 


6.  COMMENTS!  I  eagarly  await  tha  raaulta  froa  thla  SRI 
atudy.  I  have  experienced  r  great  daal  of  difficulty 
locating  any  other  afforta  at  daelgnlng  and  building  what 
I  consider  to  ba  truly  gneafully  degrading  aalf- 
rapalrlng  ayataaa.  Moat  »f  tha  effort  In  fault-tolarant 
coaputlng  to  data  aaaaa  tu  ba  cantered  erowd  nllltary 
ayataaa,  or  even  aoraao,  iround  apace  exploration  ayataaa, 
Thla  typically  dlctataa  tliat  a  fixed  aawxnt  of  coaputlng 
power  ba  aade  available  a':  all  tlaae;  lienee,  tha  lack  of 
action  aroixid  fall-aoftly  ayataaa.  Of  couraa,  by 
providing  fault  tolaranca  through  graceful  degradation, 
vary  aubatantlal  eoat  aavlnga  can  ba  raalltad  over  tha 
"redundant"  eathodJ.  In  addition  to  allowing  tha  ayataa’a 
performance  to  dagraila  In  tha  preaance  of  faulta,  wa  have 
choaan  not  to  guarantee  lntraprocaaa  Integrity.  Tha  comb¬ 
ination  of  thaaa  two  eoncaaalona  haa  allowed  ua  to  daalgn 
a  vary  aconoalcal  fault-tolarant  tlaa-aharlng  ayatae, 

Thera  la  little  doubt  that  tha  anticipated  degradation J 
will  ba  quite  acceptable  for  a  wide  range  of  application. 
Tlia  lack  of  lntraproeaae-lntagrlty  guarantaaa,  however, 
will  ba  a  Halting  factor  In  expanding  thla  architecture 
Into  other  araaa.  Of  couraa,  hardware  provlalona  could  ba 
added  to  guarantee  Introproceea  integrity,  and  tha 
reaultant  ayatea  would  etlll  ba  wore  aconoalcal  than  neat 
other  fault-tolarant  ayataaa.  A  acre  proalalng  approach, 
and  one  which  we  will  wdotitadly  explore  In  tha 
raaaonably  near  future,  la  to  leave  tha  hardware  aa  la  and 
run  critical  progrtaa  twice  on  two  different  proceaaora. 
Thla  will  allow  tha  ayatea  coat  to  raaaln  vary  low,  and 
will  alao  allow  lntraprocaaa  Integrity  guarantaaa.  Thua, 
only  thoaa  procaaaae  that  that  naad  th’a  guarantee  will 
have  to  pay  for  thla  added  feature.  A  final  aapeet  of  tha 
PRIME  architecture  that  ebould  ba  lnvaatigatad  la  whathar 
It  can  acre  economically  provide  a  guaranteed  coaputlng 
power  In  aoaa  an vl momenta  than  can  ba  provided  by  a 
"redundant”  ayatea.  It  can  ba  overbuilt  by  an  aovnt 
efficient  to  guarantee  that  lta  degraded  condition  la 
powerful  enough  to  handle  tha  nacaaaery  coaputlng,  with 
background  p~w«r  available  moat  of  tha  tlae. 


SURVEY  OF  FAULT-TOLERANT  COMPUTING  SYSTEMS 


Jacques  J,  De  lama  re,  Electronlque  Marcel  Dassault 
(E.M.D.),  55,  qusl  Carnot,  92  -  Saint-Cloud  France,  June 
1972 


1.  IDENTIFICATION 

1.1.  NAME:  MECRA  (Maquette  Experimental  de  Calculateur 
a  Reconfiguration  Automatlque) . 

1.2*  RESPONSIBILITY:  E.M.O.  (Electronlque  Marcel 
Oasseult) . 

1.3.  SUPPORT:  Support  hes  three  sourcea;  O.G.R.S.T. 
(Delegation  Cenersle  a  la  Recherche  Sclentl flque)  with 
preliminary  ctudlei;O.R,M.E.  (Direction  dea  Recherchua  et 
dea  Moyena  d'Essals)  with  realization  of  MECRA  project; 
E.M.O.  (Electronlque  Marcel  Dassault)  In  ejeh  case. 

1.4.  PARTICIPANTS:  Jacquea  J.  Delamare,  Cerard  Germain, 
Jean-Claude  R.  fharpentler,  all  of  E.M.O.,  end  four 
researchers  from  "Centre  de  Calcul  Numerlque  de  Toulouae". 

1.5.  START:  May  1970 

1.6.  COMPLETION:  July  1972,  thla  consists  of  a 
demonstration  of  fault  tolerance  and  reconfiguration 
capabilities.  Evaluation  of  reliability  performance  Is 
expected  to  be  In  Autumn  1972, 

1.7.  BIBLIOGRAPHY:  "The  MECRA:  ■  Self  Reconflgurable 
Computer  for  Highly  Relleble  Process",  IEEE  vol  C-20  no. 
11,  pp.  1382-1388,  Nov.  1971,  A  report  alao  due  end  of 
1972. 


2.  MOTIVATION 

2.1.  PURPOSE:  The  ayatem  was  conceived  for  research  In 
fault-tolerant  computer  architecture,  feasibility,  and 
reliability  evaluation.  The  Idea  for  further  developomnt 
la  a  real-time  medlum-Bized  computer  for  aircraft. 

2.2.  PHYSICAL  1NV1R0NMENT:  System  operates  In  EMD 
laboratories. 

2.3.  COMPUTING  ENVIRONMENT:  A  Single  peripheral  allows 
communication  with  (£CRA. 

2.4.  COMPUTtNC  OBJECTIVES;  Main  objectives  of  the 
ptoject  were  not  computing  objectives.  However  addition 
and  multiplication  are  performed  with  11  decimal  digits 
plus  Sign  operands.  Complete  addition  needs  less  than  300 
mlcrosec.  Such  delays  relate  to  the  cycle  time  of 
microprogram  memorv  (1  mlcrosec),  to  response  time  of 
discrete  circuits,  to  unused  time  Intervals  In  each 
mlcrolnatructlor  cycle,  (allowing  hardware  modifications), 
and  lastly  by  the  microsoftware  package  (allowing 
reconfigutatlon) . 

2 , j ,  RELIABILITY  OBJECTIVES:  Practical  experience  and  a 
concrete  basis  for  evaluation  such  ss: 
reliability  gain  with  different  kinds  of  redundancy, 
hardcore  contribution  In  failure  probabilities, 
hardcore  contribution  with  different  architectures, 
reliability  gain  with  reconfiguration, 
coat  Increase  In  control  with  reconfigurability, 
lost  time  due  to  reconfiguration  (during  and  after), 
hardcore  response  time  with  respect  to  computing  time. 
Theae  rellabi.lty  objectives  were  only  of  Interest  for 
high  probabilities  of  succesa  (probabilities  higher  than 


2.6,  DYNAMIC  VARIABILITY:  Computing  speed  but  not 
accuracy  may  degrade  with  reconf lguretlon  (201  maximum). 
Performance  cannot  be  exchanged  for  Increased  reliability 
such  as  :  two  processors  each  one  having  its  own  Job, 
switch-d  to  parallel  processing  on  the  esme  )ob  and 
checking  ope  another. 

2.1.  PE**ALTIES:  Pens] ties  from  faulty  operation  can  be 
of  several  kinds:  /Loss  of  time  due  to  recovery  processes, 
les  ned  performmee  sfter  self-reconfiguration,  loss  of 
service./  Manual  Interventions  have  not  been 
Investigated,  but  will  be  necessarily  Improved  ss  a 
coneeq  ence  of  self-testing  and  self-healing  capabilities 
of  MECkA. 

/SRI  note:  The  text  enclosed  in  slashes  is  an  SRI 
paraphrase  of  the  original  survey  response./ 

2.8,  CONSTRAINTS:  Circuitry  size  might  not  exceed  four 
tines  the  size  of  the  equivalent  irredundent  computer. 
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4.7.  IMPLICATIONS:  tU.vy  rxllanc*  1*  pi «c*il  on  pirlodlc 
eh.ckln,  of  h.rdw.r.  r.th.r  than  concurrant  eh.ckln*. 

Thua,  tha  ability  to  lnjact  faulta  Into  tha  approprlata 
araaa  haa  baan  a  difficult  raqulraaant  plaoad  on  all  of 
tha  h.rdw.r.  daalfnara,  Tha  aoat  notabla  aoftwara 
raqulraaant  lapoaad  by  tha  baalc  daalpn  la  tha  daar 
dlTlalon  of  tha  oparatlnp  ay a tan  into  thraa  parta,  ona  of 
which  can  b«  furnlahad  by  a  uaar.  Tha  only  alsnlfleant 
raqulraaant  placed  on  a  naar  la  that  ha  nuat  ba  awara  that 
no  lntraprocaaa  lntatrlty  chacka  ara  nada  (juat  Ilka  In 
all  currant  tlaa-aharln?  ayataau). 


5.  CONCLUSIONS 

5.1.  STATUS:  Tha  daal|n  of  PRDPi  la  about  95  parcont 
coaplatad,  and  lnplasantatlon  haa  bagun  on  both  the 
hardware  and  aoftwara.  Tha  flrat  varalon  capable  of 
raconfl|urln|  In  tha  preaent  of  a  failure  ehould  ba 
running  by  Auguat,  1973. 

5.2.  EXPERIENCE:  Tha  main  conclualon  that  tha  raapondant 
can  make  regarding  tha  daalgu  of  PRIME  la  that  by  aomawhat 
Halting  tha  goal  of  tha  PRIME  ayaten.  It  waa  poaalbla  to 
create  a  ayataa  that  ahould  exhibit  excellent  fault- 
tolarant  charectarlatlca  at  a  much  lowar  lncraaantal  coat 
than  that  of  any  other  fault-tolerant  ayetan  known  to  him. 

5.3.  FUTURE:  Tha  near  future  will  ba  devoted  to  building 
PRIME.  After  that,  evaluation  and  ttnlng  will  taka  place 
with  connection  to  tha  Arpanet  very  likely, 

5.4.  ADVANCES:  It  aeama  that  tha  moat  algnlflcant 
development  that  would  aid  tha  PRIME  ayatem  would  ba  tha 
availability  of  a  ganaral-purpoee,  ealf-chacklng 
procaaaor.  Since  100  percent  t  alf-checkablllty  la 
extremely  difficult  to  daalgn  Into  a  procaaaor,  tha  beat 
couraa  of  action  hare  aaama  to  ba  to  welt  for  LSI 
procaaaora  of  aufflcleot  power  to  be  built.  Thaea 
procaaaore  ehould  ba  ao  lnaxpenalve,  compared  to  tha  raat 
of  tha  hardware  coat,  that  running  two  of  them 
alnultaneoualy  and  comparing  outputa  ahould  ba  a  very 
attractive  procedure  economically.  In  fact,  tha  currant 
procoialng  clamant  in  PRIME  could  be  broken  Into  eevaral 
eubproceaeora:  one  for  cowmunlcatlona,  ona  for  tha  dlak 
controller,  ona  for  tha  terminal  controller,  two  for  the 
Target  Machine,  etc.  Probably  only  tha  Target  Machine 
procaaaor  would  have  to  ba  duplexed  becauaa  tha  othara  can 
have  Independent  chacke  on  tha  validity  of  thalr  raaulta. 
With  thle  procedure,  lntraprocaaa  Integrity  would  ba 
poaalbla  at  an  lnelgnlflcast  Incremental  coat. 


6.  COMMENTS:  I  aagvrly  await  the  raaulta  from  thle  SRI 
etudy.  1  have  experienced  a  great  deal  of  difficulty 
locating  any  other  effort.’  at  deelgnlng  and  building  what 
1  conaldar  to  ba  truly  gtecafulle  degrading  ealf- 
rapalrlng  ayatama.  Moat  of  tha  niton  In  fault-tolarant 
computing  to  date  aaama  to  ba  can'.atod  around  military 
eyetema,  or  even  moraao,  around  apace  exploration  ayatama. 
This  typically  dletataa  that  a  fixed  anoint  of  conputlng 
power  ba  made  available  at  all  tlmaa;  hence,  tha  lack  of 
action  around  fell-aoftly  ayatama.  Of  couraa,  by 
providing  fault  tolerance  through  graceful  degradation, 
very  aibatantlal  coat  aavlnga  can  ba  raellrad  over  tha 
"redimdant1’  method:’.  In  addition  to  allowing  tha  ayetam'a 
performance  to  degrade  In  tha  praaanca  of  faulta,  we  have 
choean  not  to  guarantaa  lntreproceaa  Integrity,  The  comb¬ 
ination  of  thaea  two  eoncaaalona  haa  allowed  ua  to  daalgn 
a  vary  economical  fault-tolerant  tiaa-aharlng  ayatam. 

There  la  little  doubt  that  tha  anticipated  dagradetlona 
will  ba  quit*  acceptable  for  a  wide  ranga  of  appllcatlona . 
The  lack  of  lntraprocaae-lntagrlty  guarantees,  however, 
will  ba  a  limiting  factor  In  expanding  thla  architecture 
Into  other  araaa.  Of  courea,  hardware  provlelona  could  ba 
added  to  guarantaa  lntroproeaaa  Integrity,  and  tha 
raaultant  ayatem  would  atlll  ba  more  economical  than  moat 
other  fault-tolarant  ayatama.  A  more  promlelng  approach, 
and  ona  which  ve  vlll  wdoubtadly  explora  In  the 
reaaonably  near  future,  la  to  leave  the  he-.jwera  aa  la  and 
run  critical  programa  twice  on  two  dlffarent  procaaaora. 
Thla  will  allow  tha  ayataa  coat  to  remain  very  low,  and 
will  alao  allow  lntraprocaaa  Integrity  guarantaaa.  Thua, 
only  thoea  procaeeaa  that  that  need  thla  guarantaa  will 
have  to  pay  for  thle  added  feature,  A  final  aapact  of  the 
PRIME  architecture  that  ahould  lnvaatlgatad  la  whether 
It  can  more  economically  provide  e  guaranteed  computing 
power  In  aoma  anvironmenta  than  can  ba  provided  by  a 
"redundant"  ayatem.  It  can  ba  overbuilt  by  an  ameut 
sufficient  to  guarantee  that  lta  degraded  condition  la 
powerful  enough  to  handla  tha  nacaaaary  conputlng,  with 
background  power  available  moat  of  the  time. 


SURVEY  Or  FAULT-TOLERANT  COMPUTING  SYSTEMS 


Jscques  J.  Delaware,  Electronique  Marcel  Dassault 
(E.M.D.),  55,  qusi  Carnot,  92  -  Saint-Cloud  France,  June 
1972 


1.  IDENTIFICATION 

1.1.  NAME:  MECRA  (Maquette  Experimental  de  Calculsteur 
a  Reconfiguration  Automatique) . 

1.2.  RESPONSIBILITY :  E.M.D.  (Electronique  Marcel 
Dassault) , 

1.3.  SUPPORT:  Support  haa  three  sources;  D.C.R.S.T. 
(Delegation  Genersle  a  la  Recherche  Scientlfique)  with 
preliminary  studhijD.R.M.E.  (Direction  dee  Recherchea  et 
des  Moyena  d'Essals)  with  realization  of  MECRA  project; 
E.M.D.  (Electronique  Marcel  Dassault)  in  each  case. 

1.4.  PARTICIPANTS:  Jacques  J.  Delamare,  Gerard  Germain, 
Jean-Claude  R,  Charpentier,  all  of  E.M.D.,  and  four 
researchers  from  "Centre  de  Calcul  Numerique  de  Toulouse". 

1.5.  START:  May  1970 

1.6.  COMPLETION:  July  1972,  thla  consists  of  a 
demonstration  of  fault  tolerance  and  reconfiguration 
capabilities.  Evaluation  of  reliability  performance  ia 
expected  to  be  in  Autumn  1972. 

1.7.  BIBLIOGRAPHY:  "The  MECRA:  s  Self  Reconfigurable 
Computer  for  High1*  Reliable  Froceas",  IEEE  vol  C-20  no. 
11,  pp.  1382-1388,  Nov.  1971.  A  report  alao  due  end  of 
1972. 


2.  MOTIVATION 

2.1.  PURPOSE:  The  system  was  conceived  for  research  in 
fault-tolerant  computer  architecture,  feasibility,  and 
reliability  evaluation.  The  idea  for  further  development 
is  a  real-time  medium-sized  computer  for  aircraft. 

2.2.  PHYSICAL  INVIRONMENT:  System  operates  in  EMD 
laboratories . 

2.3.  COMPUTING  ENVIRONMENT:  A  single  peripheral  allows 
communication  with  MECRA. 

2.4.  COMPUTING  OBJECTIVES:  Main  objectives  of  the 
project  were  not  computing  objectives.  However  addition 
and  multiplication  are  performed  with  11  decimal  digits 
plus  sign  operands,  Complete  addition  needs  less  than  300 
microsec.  Such  delays  relate  to  the  cycle  time  of 
microprogram  memory  (1  microsec),  to  response  time  of 
discrete  circuits,  to  unused  time  intervals  in  each 
microinstruction  cycle,  (allowing  hardware  modifications), 
and  lastly  by  the  microsoftware  package  (allowing 
reconfiguration) . 

2.5.  RELIABILITY  OBJECTIVES:  Practical  experience  and  a 
concrete  basis  for  evaluation  such  as: 

reliability  gain  with  different  kinds  of  redundancy, 
hardcore  contribution  in  failure  probabilities, 
hardcore  contribution  with  different  architectures, 
reliability  gain  with  reconfiguration, 
coat  increase  in  control  with  reconfigurability, 
lost  time  due  to  reconfiguration  (during  and  after), 
hardcore  response  time  with  respect  to  computing  time. 
These  reliability  objectives  were  only  of  interest  for 
high  probabilities  of  success  (probabilities  higher  then 


2.6.  DYNAMIC  VARIABILITY:  Computing  speed  but  not 
accuracy  may  degrade  with  reconfiguration  (201  maximum). 
Performance  esnnot  be  exchanged  for  Increased  reliability 
such  as  :  two  processors  each  one  having  its  own  job, 
switched  to  parallel  processing  on  the  same  Job  and 
checking  one  another. 

2.7.  PENALTIES:  Penalties  from  faulty  operation  esn  be 
of  several  kinds:  /Loss  of  time  due  to  recovery  processes, 
lessened  performance  after  self-reconfiguration,  loss  of 
service./  Manual  interventions  have  not  been 
investigated,  but  will  be  necessarily  improved  as  a 
consequence  of  self-testing  and  self-healing  capabilities 
of  MECRA. 

/SRI  note:  The  text  enclosed  in  slashes  is  an  SRI 
paraphrase  of  the  original  survey  response./ 

2.8.  CONSTRAINTS:  Circuitry  size  might  not  exceed  four 
times  the  size  of  the  equivalent  irredundant  computer. 


3.  DESCRIPTION 
3.1  ARCHITECTURE 

3.1.1.  CONFIGURATIONS 

3. 1.1.1.  INTERCONNECTIVITY:  See  IEEE  paper.  The  basic 
configuration  is  a  microprogrammed  nonoprncesaor  with  a 
bus  architecture.  A  restriction  can  be  seen  here  since 
addresses  are  binary  coded,  whereas  data  are  Decimal 
Hamming  coded.  This  has  no  importance  for  the  purpose  v f 
the  project,  but  would  not  have  been  used  on  a  prototype. 

3. 1.1.2.  RANGE:  Control  Unit  Configuration: 


Maximim 

4  counters 

3  spare  counters 
8  registers 

4  spare  registers 

3  multiplication  processors 
2  addition  processors 

4  'and  *  logic  processors 
4  'or'  logic  processors 

4  'exclusive  or'  processors 
4  'inverter'  blocks 


Minimum 

3  counters 
0  apare  counts rs 
6  registers 
0  spare  registers 
1  multiplication  processor 
1  addition  processor 
3  or  2 
3  or  2 
3  or  2 
3  or  2 


Note:  Any  logic  function  can  fail  completely  and  can  be 
reconfigured  with  three  other  functions.  In  several  casea 
a  failed  logic  function  can  be  reconfigured  with  only  two 
other  function, 

M-mory  configuration:  Three  memory  blocks  -  4  K  16-bit 
words.  Each  memory  block  has  its  own  addrcs*  decoder 
circuits.  At  each  memory  cycle  a  48-bit  word  is  read  or 
written;  this  word  contains  two  identical  words  of  24  bit* 
each,  so  that  any  one  of  the  three  blocks  can  be  declared 
void  and  the  computer  still  runs  if  the  other  two  operate 
properly.  Efficiency  of  address  error  detection  reaches 
502  on  each  memory  block.  After  any  read  restore  cycle, 
each  eight-bit  byte  (6  bytes)  is  checked  and  is  switched 
or  not  on  busses.  Then  error  detection  efficiency  is  502 
with  instructions  or  microinstruction  (if  there  is  only 
one  erroneous  bit)  and  1002  with  data  (if  tucre  la  one  or 
two  erroneous  bit). 


3,1,2  EXECUTIVE 

3. 1.2.1,  MOOES:  MECRA  is  a  monoprocessor. 

3. 1.2. 2,  SOFTWARE:  There  are  three  working  modes  on  the 
computer:  user  mode,  test-diagnosis  mode,  decision  and 
reorganization  mode. 

a)  In  the  USER  mode  the  computer  executes  the  user 
program. 

b)  The  TEST-DIAGNOSIS  mode  is  set  in  motion  in  two 
different  ways  to  which  two  different  programs  correspond. 
The  first  is  set  in  motion  by  interrupts  when  a  failure 
has  been  detected  by  hardware  checkers.  The  goal  of  this 
program  is  to  localize  precisely  where  the  failure 
occured.  The  second  program  is  set  in  motion  periodically 
and  its  purpose  is  to  teat  the  computer  with  the  data 
configurations  which  reveal  failures  beat.  This  program 
allows  detection  of  the  errors  which  cannot  be  detected  by 
the  hardware  checkers  (i.e.  an  erroi.eous  data  with  correct, 
encoding).  These  two  program*  update  a  atatus  table  which 
contains  the  st&t’is  of  computer  components  (failed  or  not, 
number  of  transient  failures).  They  also  decide  to  stop 
the  computer  when  certain  catastrophic  failures  occur  or 
to  set  In  motion  the  decision  and  reorganization  mode, 

c)  in  th*  (DECISION  ANO  REORGANIZATION  mode,  a  program 
analyzes  the  atatus  word  (ia  the  status  table)  of  the 
component  in  which  one  of  the  two  test- diagnosis  programs 
has  detected  a  permanent  failure  and  it  decides  either  to 
reconfigure  or  to  stop  the  computer. 

3.2,  FAULT  TOLERANCE 

3.2.1,  FAULTS  TOLERATEO:  Any  single  fault  is  tolerated 
in  memories,  arithmetic  and  logic  units  (since  they  are 
mounted  in  a  duplex  scheme)  or  in  logic  units  (quadded 
redundancy).  Any  error  detected  on  the  busses,  switches 
the  MECRA  to  interrupt  programs,  while  all  writing  in 
memories,  registers  or  counters  is  inhibited.  Multiple 
errors  can  also  be  tolerated  in  number  of  caaea.  Multiple 
errors  can  lesd  to  repair  or  to  loss  of  service  as  said 
above  (2.7.), 

3.2.2,  FAULTS  NOT  TOLERATED:  Faults  not  tolerated 
include  errors  in  the  main  control  circuit,  which  leads  to 
s  design  with  an  increased  degree  of  ml croprog reaming  and 
minimised  control  circuits.  Also  not  tolerated  are 
errors  undetected  at  the  memory  output.  Power  supply 
failures  have  not  been  investigated  in  MECRA. 


3.2.3,  TECHNIQUES:  One  of  the  goals  of  MECRA  is  an 
investigation  of  as  many  fault-tolerance  techniques  as 
possible,  such  as  triple  modular  redundancy,  qusdded 
redundancy,  duplex  redundancy  at  very  low  level  (clock) 
and  higher  level  (memories  and  arithmetic  circuits), 
random  redundancy  (counters,  registers),  error  detecting 
codes  (Hamming  d  *=  3)  and  parity  bit,  repetition, 
rollback,  reconfigurati jn  with  removal  without 
replacement,  reconfiguration  with  replacement,  diagnosia  - 
stand-alone,  preventive  and  emergency,  local  protections 
of  process  and  data.  These  techniques  are  used  statically. 

It  does  not  seem  possible  to  describe  these  techniques  in 
detail  in  thia  paper,  since  it  would  require  a  description 
of  the  whole  computer.  Other  techniques  were  also 
investigated  but  not  used  on  MECRA,  such  as  stopping  the 
computer  during  noisy  periods,  snd  control  of  correct 
microprogram  linking. 

3.3.  NOVELTY:  When  the  project  started,  two  ideas 
unusual  in  the  literature  were  employed  in  MECRA:  address 
decoder  redundancy  in  memories  so  ss  to  separate  address 
errors  and  data  errors,  aingle-error-free  hard-con?. 

3.4.  INFLUENCES:  A  synthesis  of  efforts  which  came 
almost  exclusively  from  the  U.S.A,  -  universities, 
laboratories,  and  research  institutes. 

3.5,  HARD-CORE:  This  is  defined  as  a  circuit, 
interconnecting  several  redundant  functions,  whatever  its 
own  redundancy  level  (it  is  a  relative  concept). 


4.  JUSTIFICATIONS 

4.1.  RELIABILITY  EVALUATION:  Reliability  is  not 
demonstrated,  it  is  computed,  in  two  steps  using  a  model. 
The  first  step  concerns  analysis  and  drawing  a  network 
model,  the  second  step  concerns  random  failure  assignment 
into  the  model.  After  a  great  number  of  trials,  the 
program  furnishes  results  (e.g.  curves,  marginal 
probabilities. . ,) . 

4.2.  COMPLETENESS  OF  EVALUATION:  Program  evaluation  is 
now  being  tested. 

4.3.  OVERHEAD:  Approximately  602  to  702  of  total  system 
resources  are  demoted  to  fault  tolerance  (same  percentage 
for  ioeic,  cost,  and  time). 

4.6,  CRITICA1 1 TIES :  Use  of  decimal  coded  characters 
seems  not  well*  fitted  to  fault-tolerant  computers.  Thia 
change  could  result  in  great  savings  in  design.  Other 
points  are  not  critical. 

4.7.  IMPLICATIONS:  The  basic  design  assumes  low-level 
integrated  circuits,  wtih  a  very  small  number  of  different 
circuits . 


5.  CONCLUSIONS 

5.1.  STATUS:  The  system  is  now  operating  and  will  be 
delivered  in  July  72,  evaluation  will  follow  during 
October  snd  November. 

5.2,  EXPERIENCE:  Everything  is  possible,  except,  perhaps 
a  sufficiently  low  cost,  and  reliable  packaging  snd  wiring 
of  components.  Note  that  LSI  would  put  problems  to 

fault -tolerant  computers  because  they  need  more  pins  to 
check  redundant  functions  before  connecting  all  together. 
This  would  probably  lead  to  simultaneous  i-e  of  LSI,  MSI 
and  f-msll  scale  integrated  circuits.  Com-,onen>. 
manufacturers  have  not  yet  taken  into  a r count 
fault-tolerance  constraints,  but  they  vi.ll  probably  do  so 
soon. 


5.3.  FUTURE:  First  prototype  is  projected  1976  -  1977, 
Current  computer  ia  projected  1980,  Use:  Missiles, 
aircraft,  real-time  monoprocessors . 

5.4,  ADVANCES:  Oifferent  fault  tolerant  computers  can  be 
roughly  compared  in  rerms  of  reliability  versus  mission 
time;  but  this  will  fall  back  to  evaluations  of  components 
and  wiring  MTBF,  Such  data,  estimated  by  constructors,  do 
not  aeem  to  give  a  sufficient  common  basis  for 
evaluations.  Theoretical  and  conventional  data  on 
component  MTBF  seem  to  be  needed  for  accurate  comparisors 
among  different  fault-tolerant  computers. 
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SURVEY  OF  FAULT -TOLERANT  COMPUTING  SYSTEMS 

L.  J.  Koczela,  North  American  Rockwell  Corp. 

3370  Mireloma  Avenue,  Anaheim,  California  92803,  May  1972 


1.  IDENTIFICATION 

1.1.  NAME:  A  Three  Failure  Tolerant  Computer  Syatem 

1.2.  RESPONSIBILITY:  Electronic!  Group,  North  American 
Rockwell  Corp. 

1.3.  SUPPORT:  Manned  Spececraft  Center,  NASA 

1.4.  PARTICIPANTS:  L.  J,  Koczela,  J.  Juriaon,  D.  Broaius 
-  North  American  Rockwell;  P.  Sollock  -  NASA, 

1.5.  START:  1/1/70 

1.6.  COMPLETION:  1/1/71  (design  concept) 

1.7.  BIBLIOGRAPHY:  A  Three  Failure  Tolerant  Computer 
System,  IEEE  Trens.  on  Computers,  November  1971 


2.  MOTIVATION 

2.1.  PURPOSE:  Real-Time  Central  Guidance  and  Control 
Computer 

2.2.  PHYSICAL  ENVIRONMENT:  Spacebome 

2.3.  COMPUTING  ENVIRONMENT:  Hie  computer  ayatem  interacts 
with  avionics  subsystems  via  a  multiplexed  data  bus. 

2.4.  COMPUTING  OBJECTIVES:  30,000  words  of  memory; 

500,000  operetions/aecond  apeed 

2.5.  RELIABILITY  OBJECTIVES:  Must  tolerate  firat  two 
failures  with  no  dagradation  in  performance  and  third 
failure  with  no  degradation  in  aefety. 

2.6.  DYNAMIC  VARIABILITY:  Third  failure  could  heve  leas 
computational  capacity. 

2.7.  PENALTIES:  Would  require  manual  intervention  with 
possible  loss  of  life. 

?.3.  CONSTRAINTS:  No  physical  constraints  but  a  relative 
weighting  of  importance  between  physical  perametera, 

2.9.  TRADEOFFS:  Size,  weight  and  power  least  important. 


3.  DESCRIPTION 

3.1.  ARCHITECTURE 

3.1.1.  CONFIGURATIONS 

3. 1.1.1.  INTERCONNECTIVITY:  Four  redundant  computers 
interconnected  by  four  voter  switches  at  their  I/O 
channels. 


3,1. 1,2.  RANGE:  2-6  CPUs,  no  restrictions  on  word 
length. 


3. 1.1.3.  CAPABILITY:  500,000  operations/second 

3.1.2.  EXECUTIVE 

3. 1.2.1.  MODES:  Hie  executive  may  operate  the  redundant 
computers  in  r-ny  modes  of  operation:  non-redundent 
independe'-c  computers,  multi-prograomed,  multi-computer, 
and  various  combinations  of  redundancy  such  as  comparison, 
voting,  etc. 

3. 1.2.2.  SOFTWARE:  Software  control  is  equally  diatributed 
among  the  redundant  computers  -  no  central  control  exiata, 

3.2.  FAULT  TOLERANCE 

3.2.1.  FAULTS  TOLERATED:  Any  3  faults.  A  fault  can  range 
from  a  single  circuit  element  to  a  complete  module  such  aa 
e  CPU  failing.  A  failure  haa  no  effect  on  ayatem  behavior. 
The  system  actually  tolerate  more  than  three  faulta  of 
many  different  types  but  it  will  tolerate  st  least  any 
three  faulta. 

3.2.2.  FAULTS  NOT  TOLERATED:  Software  faults  that  ere  not 
caught  in  debugging, 

3.2.3.  TECHNIQUES:  The  tachnlque  used  is  replication  of 
hardware  with  quedrupla  redundancy.  Computation*  are 
performed  redundently  end  reconflguretion  la  accomplished 
without  removal  or  replacement  after  failure  detection  by 
voting. 


3.3.  NOVELTY:  Through  the  redundant  use  of  adaptive 
voters  operating  on  the  input/output  of  redundant 
computera,  any  three  failure  can  be  tolerated. 

3.4.  INFLUENCES:  None 

3.5.  HARD-CORE:  No  hard  core  exlcts. 


4.  JUSTIFICATION 

4.1.  RELIABILITY  EVALUATION:  Extensive  fault  simulations 
have  been  successfully  performed, 

4.2.  COMPLETENESS  OF  EVALUATION:  It  is  impossible  to 
verify  a  design  goal  of  100  percent  confidence. 

4.3.  OVERHEAD:  For  triple  failure  tolerance,  about  80Z, 
Iesa  for  lower  failure  tolerance. 

4.4.  APPLICABILITY:  To  many  critical  real-time  control 
syatems,  industrial,  space  and  defense  applications. 

4.5.  EXTENDABILITY:  The  design  can  be  extended  to 
tolerate  different  numbers  of  failures,  eg.  any  two 
failures,  any  four  failures,  etc. 

4.6.  CRITICALITIES:  Requirement  for  100X  confidence  in 
tolerating  any  3  failures  is  very  critical,  lowering  to  99 
percent  or  so  would  reduce  complexity  snd  cost. 

4.7.  IMPLICATIONS:  Hardware  designers  must  insure 
independence  of  failures  at  computer  I/O  Interfaces. 

5.  CONCLUSIONS 

5.1.  STATUS:  System  design  concept  completed, 
voter-switch  detailed  design  completed,  prototype  hardware 
of  voter-switch  currently  under  development. 

5.2.  EXPERIENCE:  A  very  rigid  failure  tolerance 
requirement  can  be  met  assuring  that  a  minimum  number  of 
failures  will  be  tolerated. 

5.3.  FUTURE;  Possible  use  on  space  shuttle  program 

5.4.  ADVANCES:  A  significant  area  that  can  enhance  the 
state  of  the  art  in  designing  fault- tolerant  computers  ia 
analysis  of  failure  modes  of  components  and  computer 
subsystems  in  depth.  Another  v<*ry  important  area  is 
error-free  software. 


6.  COMMENTS:  Much  of  the  work  on  f ault-tolerant 
computers  is  dedicated  to  single  failures  at  the  gate  and 
circuit  level.  Unfortunately,  in  many  cases  this  is  not 
applicable  to  real  world  failurea  when  considering 
computers  mechanized  from  state  of  the  art  LSI  Integrated 
circuits. 
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SURVEY  OF  FAULT-TOLERANT  COMPUTING  SYSTEMS 


J.  S.  Miller,  Intermetric.  Inc. 

701  Concord  Ave.  Cambridge,  Mae*.  02138,  May,  1972 


1.  IDENTIFICATION;  The  eyetem  Is  referrad  to  only  as  the 
T^™trlCS  MultlProc*eeor  (occasionally  abbreviated 
IMP).  It  Is  eponeored  by  the  NASA  Manned  Spacecraft 
Center,  Houston,  Texas.  Personnel  participating  In  the 
deelgn,  in  addition  to  myself,  are  W.  H.  Vendever  A  L 
Kosmala,  S.  F.  Stanten,  S.  J.  Schwartz,  and  A.  Avakian.’ 
The  project  began  In  Juna,  1969,  and  has  continued  to 
dete,  except  for  a  thirteen-month  Interval  betwean  the 
original  contract  and  the  current  one,  which  euda  In  about 
two  months.  Ons  report  has  been  publiehed:  "Final 
Report— Multiproceseor  Computer  System  Study",  by  James  S. 
Miller,  Daniel  J.  Lickly,  Alex  L.  Kosmala,  and  Joseph  A. 
aponaro.  In  March,  1970.  A  second  report  le  presently  in 
preparation.  The  work  has  been  design-only;  no  hardware 
is  Involved. 


2.  MOTIVATION;  The  system  la  oriented  towards  the 
general-purpose  computational  requirements  oi  a  manned 
orbiting  space  stetlon  of  ebout  the  1980  time  period.  Its 
expected  uses  include  reel-time  atatlone  control  and 
data-acquieltlon  functions,  plus  Interactive  and  batch 
data  processing  operations.  The  performance  objectives 
are  soft,  but  a  real-time  reeponse  of  5  ns  or  better  and 
an  equivalent  of  two  million  additions  per  eecond  for  a 
three-procassor  configuration  seem  adequate. 

Because  no  hardware  Is  being  deelgned,  no  epeclflc 
reliability  figure  has  been  Imposed.  The  use  of  the 
system  for  control  of  the  apace  station  lteelf  pieces 
heavy  emphasis  upon  continued  operation,  at  reduced 
performance  in  the  presence  of  faults.  Although  we  expect 
that  temporary  outeges  of  the  computer  system  will  be 
tolerable,  our  efforte  heve  baen  directed  at  avoidance  of 
single-point  failure  modes. 

3.  DESCRIPTION:  Although  an  earlier  design  favored  the 
use  of  a  single  Internal  bus  connecting  all  modulee,  with 
caches  et  each  proceesor  Incorporated  to  dimlnieh  traffic, 
we  have  now  settled  upon  a  more  conventional  croeebar-llke 
network.  Each  processor  and  main  memory  module  possesees 
a  bus,  as  does  the  I/O  controller.  Secondary  storage  Is 
accessed  via  the  latter  unit.  Configurations  of  up  to 
eight  processors,  and  ae  few  es  one,  are  planned,  with 
nominally  three.  A  32-bit  word,  with  additional  bits 
added  for  error-detection,  has  been  chosen. 

A  full  complement  of  multiprocessing  Is  provided  by  the 
system  software.  Processes  may  be  dependent  or 
independent  of  each  other.  Eech  processor  la 
multlprogrammed  by  a  "floating"  operating  system  executed 
by  any  processor,  as  neceeeary.  Interprocess 
communication  is  eupported,  and  processes  may  field  their 
own  tterrupts  If  they  choose. 

The  Instruction  set  of  the  processor  is  designed  to 
support  the  execution  of  hlgh-order  block-structured 
languages,  such  as  Algol,  PL/I,  and  HAL,  the  last  being  an 
Intermetrics-designed  language  aleo  eponsored  by  NASA/MSC. 
The  Instruction  set  eomewhat  reeemblee  that  of  the 
Burroughs  B6700,  although  eubetantial  differences  have 
been  introduced.  It  Is  planned  that  only  high-order 
source  languages  will  be  supported.  The  system  Is 
designed  to  tolerate  "all"  faults  In  the  hardware, 
provided  that  second,  Independent,  faults  do  not  occur 
before  recovery  action  Is  complete.  Our  experience  with 
the  error-prone  discipline  of  software  restarts  on  the 
Apollo  program  hes  steered  the  design,  Compreheneive 
error-detection  fscilltlee  ere  provided  in  the  herdware , 
so  that  detection  ie  lmnedlate  and  highly  probable, 
furthermore,  eufficlent  redundant  Information  ie 
maintained  In  Independent  locations  so  that  the  operating 
system  end  hardware  capabilities  are  adequate  to  continue 
execution  of  all  processes  (subject  to  reduced  performance 
limitations)  without  explicit  pertlcipation  by  application 
software. 


Processors  3re  dually-redundant  to  provide  complete 
error-checking  capability.  Instruction  execution  is 
devised  so  that  Inputs  are  always  preserved  until 
error-free  completion  Is  signalled;  thus  re-try  la  always 
possible.  Processor  state  information  Is  maintained  in 
local  memory  which  Is  externally  accessible,  so  that 
execution  of  an  Instruction  Interrupted  by  a  processor 
fault  may  be  resumed  by  another  processor  If  re-try  proves 
unsuccessful. 

Main  memory  is  dynamically  time-multiplexed,  using 
vsrleble-slze  segments,  similar  to  the  B6700.  The 
read-write  functions  of  the  memories  ara  implemented  in  a 
way  that  enables  duplicate  storage  of  Information  when 
this  Is  specified  via  the  high-order-language.  Procedure 
code  and  other  read-only  data  will  be  resident  in 
secondary  storage,  and  thus  need  not  be  duplicated  In  main 
store.  Variable  data  may  or  may  not  be  stored  doubly;  if 
not,  the  owning  process  will  be  marked  for  termination  if 
such  data  is  destroyed  through  memory  fault.  The 
duplicate  storage  of  specified  data,  although  supported  by 
hardware,  Is  relatively  transparent  to  the  memory 
management  software;  no  epedally-deslgnated  memory 
modules  are  used. 

4.  JUSTIFICATION:  The  design  Is  relatively  complete,  but 
has  not  been  evaluated  by  any  formal  procedure.  Somewhat 
deliberately,  it  wss  bssed  on  working  architectures  to 
reduce  the  number  of  possibilities  for  unanticipated 
difficulty. 

As  mentioned  above,  processors  are  duplexed,  and  some 
Information  contslnad  In  main  mumory  Is  stored 
redundantly.  An  additional  "overhead"  for  fault-tolerance 
Is  triple-redundancy  In  certain  access-elements  of  main 
memory,  and  the  1/0  controller. 

In  our  judgement,  the  system  we  heve  designed  Is  suitable 
for  Implementation  In  any  application  where  transparency 
to  faults  and  continued  operation  are  worth  the  cost  ol 
the  redundancy.  It  Is  emphasized  that  comprehensive 
error-det*  't  on,  apart  from  recovery,  Is  responsible  for 
much  of  th'  additional  cost. 

No  claim  Is  made  that  the  syBtem  ie  tolerant  of  software 
flaws.  However,  the  lnsletence  on  use  of  high-order 
language  la  expected  to  reduce  the  probability  of  such 
errors,  both  by  making  their  commission  less  likely,  and 
by  Implementing  error-detection  features  in  the  langusges 
and  compilers.  Run-time  checking  will  be  provided  via 
hardware  features  (where  possible)  and  by 
compiler-inserted  code  to  at  least  signal  the  occurrence 
of  software  misbehavior,  before  the  effecta  can  propagate. 


5.  CONCLUSIONS:  Intermetrics  believes  that  Its 
fault— tolerant  H0L— machine  would  be  cost— competitive  with 
a  "conventional"  system  manufactured  in  the  same  quantity, 
since  the  memory  saved  by  usage  of  H0L  instructions  is 
expected  to  be  significant.  Further  development  will  be 
pursued. 


39 


SUKVLTT  OF  FAULT  TOLERANT  COMPUTER  SYSTEMS 
Donald  C.  Wallace 

Stanford  Roaaarch  Inatltuta,  Manlo  Park  Ca,  June  72 
1,  IDENTIFICATION 

1.1  NAME:COMEF-  Online  o-der  handling  ayatan 

1.2  RESPONSIBILTY;  P.C.Sarvlca  Corp.  (subsidiary  Pacific 
Coast  Stock  Exchange) 

1.3  SUPPORT:  Membsr  fines  of  PCSE 

1.4  PARTICIPANTS:  Member  flrme  of  PCSE 

1.5  START:  Contract  lar  :7  November  1967 

1.6  COMPLETION:  Syetem  accepted  -  4  December  1969 

1.7  BIBLIOGRAPHY:  Tha  moat  accurate  daarlptlon  of  the 
COMEX  la  the  final  document  lion  delivered  with  the  syatem. 
Documenta ’ 

Specification  for  data  processing  and  cooisunlcatlon 
equipment  for  Pacific  Coaat  Stoc.  Ex-hange  PC  Service 
Corp.,  1967 

Propoaal  for  Real-Time  Order  Handling  System  BBN 
#p68-DE-01,4  August  1967 

Contract  for  Real-Time  Order  Handling  Syatem  for  Ptclflc 
Coaat  Stock  Exchange  BBN /PCSE ,17  November  1967 

2.  MOTIVATION 

2.1  PURPOSE:  Real  time  odd-lot  order  execution 

2.2  PHYSICAL  ENVIRONMENT:  Ground  baaed 

2.3  COMPUTING  ENVIRONMENT:  Die  system  serves  tvo  trading 
floors,  one  In  Los  Angalea,  the  other  In  San  Francisco. 

2.4  COMPUTING  OBJECTIVES:  COMEX  la  designed  to  handle 
virtually  all  low-speed  teletype  speeds,  levels  and  codas. 
It  appears  as  a  node  on  each  of  tha  connected  broker  firms 
communication  networks  and  must  conform  to  the  line 
protocols  and  hardware  constraints  of  that  network.  The 
design  objectives  were  for  64  "nodea"  In  LA.  and  64  In 
SF.,  and  for  a  maximum  message-switching  traffic  of  25,000 
ordera/transactlons  per  day. 

2.5  RELIABILITY  OBJECTIVES:  Tha  system  was  designed  to 
provide  991+  uptime  and  with  a  no  "message  loat"  criteria. 

2.6  DYNAMIC  VARIABILITY:  The  system  la  designed  so  that 
order  entry  la  performed  In  real  time,  but  the  order 
execution  process  may  lag  an  arbitrary  parlod  of  time.  In 
operation  this  lag  never  exceeds  20  minutes  (approx.??). 

2.7  PENALTIES:  COMEX  has  various  degrees  of  degradation, 
the  ultimate  being  total  manual  oparatlon  and  execution  of 
the  orders  by  the  specialists  on  tha  trading  floors. 
Esoteric  software /hardware  malfunctlcna  could  causa 
extremly  large  manual  Intervention  pru-leme  as  the  system 
Is  really  buying  and  sailing  stock  on  the  behalf  of 
members  of  the  exchange. 

2.8  CONSTRAINTS:  The  PCSE  la  really  two  exchanges  with  two 
dlffarant  trading  floors,  one  In  lo3  Angeles  and  one  In 
San  Francisco.  For  reliability  reasons  the  system  Is 
fully  redundant.  A  PCSE  constraint  on  tha  system  was  that 
the  syatem  be  equally  spilt  between  tha  two  sites. 


3.  DESCRIPTION 

3.1  ARCHITECTURE 

3.1.1  CONFIGURATION 

3. 1.1.1  INTERCONNECTIVITY:  See  diagram  which  shows  the 
twin  IBM  360  computers  and  the  680  systems  each  of  which 
Includes  s  0EC  PDP8  computer. 

3. 1.1.2  RANGE:  The  system  Is  really  two  systems  running  In 
parallel.  It  Is  sensible  to  run  them  as  single  units  or  a 
fully  redundant  system.  Two  configurations  are  possible :- 
Non-partltionad  trading  floors: 

LA-rsmota680,  SF-local680  and  SF-360 
SF-remota680,  LA-local680  and  LA-360 
Partitioned  trading  floors: 

SF-local680  and  SF-360 
LA-local680  and  LA-  360 

3. 1.1. 3  CAPABILITY:  COMEX  consists  of  two  (2)  360/50 
computers  plus  tha  front-end  co«  unicat  Ions  systems. 


3.1.2  EXECUTIVE  and  operating  system:  COMEX  runs  under 
IBM/360  DOS  with  lta  fixed  number  of  multiprogram 
partitions  option. 

3. 1.2.1  MODES  of  operation:  The  order  execution  process 
runs  In  a  high  priority  partition  of  DOS  while  normxl 
operation  of  PC  Service  Corp.  computer  operations  are 
being  run  In  other  "foreground"  and  the  background 
partitions.  Tha  conmunl cation  process  (in  the  68G's)  Is 
dedicated  and  allows  no  other  functions. 

3. 1.2.2  SOFTWARE  organltatlon:  Basically  the  680's  do 
character  assembly  (bits),  line  protocol  Interpretation 
(answer  back,  echo,  etc...),  message  segment  assembly,  I/O 
buffering,  transmission  to  local  and  remote  360's.  The 
:60’s  do  message  switching,  code  translation,  messige 
decoding  (syntax  analysis),  order  queuing,  decoding  of 
NYSE  and  AMEX  tickers  (identify  trades),  execute  queued 
orders,  send  confirmations  to  broker  and  specialist. 

3.2  FAULT  TOLERANCE 

3.2.1  FAULTS  TOLERATED:  Essentially  tha  system  will 
tolerate  any  or  sll  failures  In  a  single  system  (i.e., 
backup  or  primary). 

5.2.2  l AULTS  NOT  TOLERATED:  Any  simultaneous  failures  in 
loth  the  primary  and  backup  ryatem  causes  loss  of 
integritry  of  the  data  flies.  This  la  considered  a 
catastrophic  event  and  some  manual  correction  and 
intervention  for  order  execution  and  notification  will  be 
needed.  (To  my  knowledge  this  has  only  occured  one.  In 
the  almost  three  years  of  operation.) 

3.2.3  TECHNIQUES: 

HARDWARE:  The  COMEX  system  Is  completely  reuundsnt 
(two  of  everything),  and  both  systems  run  In  parallel 
The  major  design  criteria  was  that  nothing  should  hep: en 
in  one  system  half  that  could  adversly  effect  the  nth, r . 
This  led  to  the  system  Interconnections  (PCU)  being 
unidlr-ctionsl  end  step-locked  in  a  "here's  a  word,  take  s 
w "  -  fashion.  All  TTY  connections  to  the  system  are  dual 
dropped  and  thare  Is  a  hardware  Interlock  to  prevent  both 
680  machines  from  outputlng  to  a  line  st  the  same  time, 

SOFTWARE:  The  software  Is  designed  to  be  very 
modular,  and  no  control  flow  exists  between  functional 
routines.  Control  flow  Is  between  the  COMEX  scheduler/ 
executive  and  each  fun.tlonsl  module.  Data  la  passed  from 
function  to  function  by  means  of  stacks  and  lists,  and 
standard  system  global  routines  are  used  to  accomplish 
tills.  Both  systems  are  actually  performing  the  entire 
order  execution  task  In  parallel  and  there  Is  really  no 
const unicstlon  between  them.  The  only  difference  la  thst 
the  "backup"  system  is  not  outputlng  transaction 
confirmations  and  order  receipt  notifications.  The  backup 
system  maintains  a  queue  of  the  last  "n”  messages  to  each 
line  in  the  syatem.  When  switch-over  occurs,  these 
messages  are  output  to  the  speclallsts/brokers  with  a  "may 
be  duplicate"  tag. 

3.3  NOVELTY:  The  Interconnection  of  the  DEC  680' s  and  the 
S/360's  la  accomplished  without  requiring  modifications  or 
additions  to  the  IBM  operating  system  or  providing 
"special"  I/O  nodules.  The  680's  (two  of  them)  have  a 
S/360  channel  equivalent  (FCU)  that  talks  to  the  IBM  2841 
disk  controler  with  the  two  channel  feature  (8100),  This 
is  the  equivalent  of  having  two  360  systems  talking  to  one 
disk  system.  This  Is  a  standard  IBM  configuration 
possibility  (though  not  supported  hy  IBM  software).  If 
the  user  Is  willing  to  accept  implementing  his  own 
read'wrlte  lock  mechanisms  there  Is  nothing  in  the  IBM 
systej  to  preclude  this  mods  of  operation.  Given  all  of 
the  above  it  Is  now  possible  to  write  a  communications 
system  strictly  at  the  user  level  using  standard  IBM  I/O 
software.  Data  Just  "appears"  on  the  disk  and  la  resd 
into  tha  360  and  Is  In  turn  wrlttsn  on  the  disk  and  Just 
"disappears".  The  dsta  from  the  680's  la  written  as  a 
sequentially  ever  growing  file,  capturing  an  entire  day's 
transactions.  This  allows  "rerunning"  a  day's 
transactions  In  resl  time  to  find  obscure  bugs. 

3.4  INFLUENCES:  After  spending  -.aver*!  .ears  working  on 
modified  or  bastard  360  Systems  and  realizing  the  effort 
leval  to  maintain  these  syeteme  given  the  fraqt«ncy  of  new 
IBM  releases,  it  seamed  Insane  to  design  s  system  that 
railed  on  any  thing  except  tha  most  rudimentary  features 
of  ths  IBM  monitor.  Ths  approach  described  has  proven 
vary  successful  In  over  three  years  of  operation.  To  my 
knowledge  no  problems  have  been  encountered  due  to  the 
monitor/  Comex  system  lnterfsce. 
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4.  JUSTIFICATION 

4.1  RELIABILITY  EVALUATION;  The  ays  ten  has  net  *nd 
exceeded  the  design  criteria  over  the  last  2  yeera  of 
operation. 

4.3  OVERHEAD;  Since  the  system  is  totally  redundant,  at 
least  half  the  ;oet  of  the  coobi  uni  cations  front  end  is  due 
to  reliability  requirementa.  The  reliability  requirements 
of  the  ayetem  probably  did  not  contribute  aigni ficantly  to 
ihe  software  design,  and  probably  helped  in  the  checkout 
end  operational  phases. 

4.4  APPLICABILITY;  The  ayetem  hea  general  epplicebili ty 
for  conn uni  cations  and  neaaage  switching  systems  where  the 
bese  computer  facility  trust  be  IBM  (for  what  ever 
reasone).  It  offers  significant  cost  savings  when 
compered  to  an  equivalent  all-IBM  equipment  configuration. 
Its  novel  interfacing  technique  allows  the  usere  to 
concentrate  on  the  applicetion  program  and  offers  long¬ 
term  savings  in  effort  by  not  having  e  modified  IBM 
operating  system.  The  ays*.sm  has  specific  applicability 
to  other  amall  or  moderete  al^ed  atock  exchanges  both  U.S. 
and  foreign. 


4.5  EXTENDABLILITY ;  There  appear  to  be  no  obvious 
extentlons  to  the  ays  ten  as  far  as  capacity  is  concerned. 
Two-  or  three-fold  Increases  in  throughput  are  possible 
whereas  factors  of  t'*n  are  out  of  the  qurstion.  Since  the 
next  obvious  exchange  automation  task  is  either  NYSE  or 
AMEX  and  the  volume  of  message  traffic  for  those  exchangf 
la  ataggerlng.  COMEX  most  certainly  has  no  real  logical 
extension  for  th— -s  situations.  Specific  experience  and 
techniques  in  de«xlng  with  automation  of  a  stock  exchange 
process  may  have  general  applicability. 

4.6  CRITICALITIES:  A  specific  goal  in  hardware  design  not 
to  exceed  the  "state  of  the  art"  was  Imposed  by  PCSE  to 
gain  assurance  of  reliability.  This  constraint  caused  the 
selection  oi  hardware  that  most  assuredly  is  obsolete  by 
today's  standards  (e.g.,  bit  serial  TTY  Interface), 
greatly  restricting  overall  I/O  capacity  (like  maybe  a 
factor  of  10). 

5.  CONCLUSIONS 

5.1  STATUS.  The  system  is  currently  handling  152  of  the 
maximum  message  switching  capacity  of  25,000  order 
transactions  per  day.  It  is  undergoing  significant 
modification  to  handle  round-lot  traffic,  which 
potentially  will  increase  load  to  501;  of  capacity  within 
the  next  18  months.  Studies  are  underway  evaluate 
high-speed  1/0  capability. 

5.2  EXPERIENCE:  Overall  system  operation  has  been  highly 
satisfactory  to  the  PCSE, 
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Werner  Ulrich,  Bell  Labs 
Naperville,  Illinois  60540  Msy  1972 


Essantially,  this  questionnaire  reprasants  the  antira  body 
of  pub  11  »nad  materiel  on  maintenance  aspects  of  ESS,  I 
have  therefore  taken  the  liberty  of  sanding  a  bibliography 
in  place  of  a  completed  quastionnaira.  You  will  notice 
that  most  of  the  articles  are  quite  brief  with  the 
exception  of  items  1  and  2  which  are  complete  descriptions 
of  the  No.  1  and  No,  2  ESS  maintenance  plan,  and  item  3 
which  ia  a  longer  article  on  a  specialized  detail  of  our 
trouble  location  manual  or  dictionary  approech. 

The  bibliography,  in  addition  to  articles  on  No.  1  and  No. 
2  ESS, contains  items  on  our  data  switching  eystem  (never 
coaaerclelly  offered,  item  5),  the  traffic  service 
position  system  (item  9),  and  a  militery  application  of 
No.  1  ESS  (item  14). 
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Capt.  L.  A.  Fry,  Space  and  Mleslle  Syetems  Organisation 
(SAMS0),  Loa  Angeles  AFS,  California,  June  1972. 

1.  IDENTIFICATION 

1.1  \AME:  Moduler  Spacecraft  Computer 

1.2  RESPONSIBILITY:  SAMS0/SYT,  Los  Angelea  APS,  Ca. 

1.3  SUPPORT:  Not  available 

1.4  PARTICIPANTS:  Pay  the on  Company,  Sudbury,  MA , , 
Ultrasystems ,  int.,  Newport  Beech,  Calif. 

1.5  START:  Project  eterted  mid-1971 

1.6  COMPLETION:  Architecture  study  completed  January 
1972.  Other  efforte  continuing. 

2  MOTIVATION 

2.1  PURPOSE:  Support  of  all  eetelllte  date  processing 
requirements 

2.2  PHYSICAL  ENVIRONMENT:  In  satellite 

2.3  COMPUTING  ENVIRONMENT:  Hardwired  to  environment 

2.4  COMPUTING  OBJECTIVES:  Approximately  200K  ope  ret  ions 
per  second.  Memory  expendable  to  basic  32  bit  word  format 
with  64K  words  of  memory. 

2.5  RELIABILITY  OBJECTIVES:  High  reliability  for  5  year 
life 

2.6  DYNAMIC  VARIABILITY:  Essentially  no  variability 

2.7  PENALTIES:  Loea  of  major  satellite  functions 

2.8  CONSTRAINTS:  25  pounds  and  30  watts. 

3.  DESCRIPTION 

3. 1  ARCHITECTURE 

3.1.1  CONFIGURATIONS:  Not  available 

3.1.2  EXECUTIVE 

3. 1.2.1  MODES:  Interruptible  but  not  e  true 
multiprocessor 

3. 1.2.2  °  .yFTVARE :  Not  yet  developed 

3.2  FmULT  TOLERANCE 

3.2.1  FAULTS  TOLERATED:  Traraient  and  permanent—  *11 
logic  types.  Also  cen  tolerc  cstestrophic  faults. 

3.2.2  FAULTS  NOT  TOLERATED:  Faults  raaulting  from  major 
phyelcel  damage. 

3.2.3  TECHNIQUES:  Replication;  coding;  repetition  end 
rollback;  and  reconfiguration.  Techniques  used 
etatlcelly  end  dynamically. 

3.3  NOVELTY:  Extensive  dynamic  redundancy 

3.4  INFLUENCES:  Not  available 

3.5  HARD-CORE:  Configuration  Control  Unit  la 
triply-moduler-redundent ,  controlling  ell  retries  and  roost 
reconfigurations. 

4.  JUSTIFICATION:  Not  nvallable 


5.  CONCLUSIONS 

5.1  STATUS:  Performing  interpretive  simulation 

5.2  EXPERIENCE:  Architecture  very  suitable  for  Intended 
application 

5.3  FUTURE:  Not  eveileble 

5.4  ADVANCES:  Not  available 


6.  COMMENTS:  Fault-tolerent  computers  can  maka  a  major 
contribution  to  long  duration  space  mlaalone. 
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abstract 


A  hierarchical  design  framework  for  fani*  ►  i 

considered  here  The  (  m  «  -tolerant  computing  systems  is 

here.  The  intrinsic  flexibility  and  dynamic 

reconfigurability  which  result  greatlv 

preatly  enhance  the  effectiveness  of 
y  tern  operation  and  system  development. 


introduction 

Recent  efforts  in  developing  large  computing  system  have  led  to  the 
cone  ns  on  that  carefully  conceived  Internal  oyster  structure  1. 
beneficial  to  the  whole  development  process  It  i,  th 

::  r  wer  that  -  —  -  especially  hem;:;  i;s:r"tenti°" 

development  and  operation  of  fault-  f«i 

there  Is  continuous  correct  Perfo™  7‘  C0"’PUt1"8  Sy5te”'  *"  ^ 

correct  performance  despite  Internal  malfunctions. 

A  framework  is  considered  here 

for  fault  tolerance  to  be  applied  at  e  h  f  3  ^  ^  °f 

hierarchy,  when  and  where  most  effective'  ViTlT  ^  3 

rrective.  This  hierarchical  framework 
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permits  fault  tolerance  to  be  achieved  at  low  cost,  especially  in 
systems  v;ith  some  real-time  leeway.  Various  Implications  are  examined. 
The  framework  Is  applicable  primarily  to  designs  for  new  systems.  It  is 
also  suitable  for  the  software  of  some  existing  systems.  Finally,  a 
problem  is  considered  which  is  greatly  simplified  by  employing  the 
hierarchical  framework.  This  is  the  massive-transient  recovery  problem, 
in  which  arbitrarily  many  unknown  faults  may  have  occurred. 

FAUI.T-TOLER,  E  TECHNIQUES 

A  variety  of  techniques  exists  for  increasing  system  fault-tolerance 
/ 1—2 / •  TVo  basic  types  of  techniques  are  usually  found  in  execution, 
static  aud  dynamic.  STATIC  techniques  involve  preplanned  actions  with 
no  changes  in  the  operating  environment  or  in  the  flow  of  control 
("fault-masking"  via  coding,  replication  with  voting,  etc).  DYNAMIC 
techniques  involve  detection  and  diagnosis  of  faults,  followed  by  a 
non-trivial  corrective  action.  Examples  include  repetition  (e.g., 
rolling  back  the  entire  system  to  an  earlier  valid  state)  and 
reconfiguration  with  or  without  replacement  by  spares  (e.g,,  removing  or 
working  around  faulty  units,  and  either  substituting  spares  or  accepting 
a  degradation  in  capacity).  In  certain  cases,  human  intervention  is 
useful.  Pre-execution  techniques  are  also  useful,  e.g.  proofs  of 
correctness  of  programs  and  design.  Note  that  static  techniques  may  be 
found  in  dynamic  usage,  e.g.,  replication  used  only  for  particular 
processes  '.c  certain  times  (see  below).  Similarly,  conceptually  dynamic 
techniques  may  appear  in  static  usage  (e.g.,  instruction  retry  in  which 
all  eventual  results  are  buffered  making  rollback  trivial). 

LEVELS  OF  STRUCTURE  AND  DYNAMIC  RECONFIGURATION 

Many  design  approaches  assume  that  essentially  all  single  faults  are 
equally  critical.  In  reality,  certain  faults  may  be  far  more  critical 
than  others.  Thus  a  system  architecture  is  desired  in  which  fault 
tolerance  techniques  may  vary  in  time  and  space,  depending  on  current 
usage  and  on  the  criticality  of  the  errors  which  might  otherwise  result. 

As  used  here,  "dynamic  reconfiguration"  implies  alteration  during 


44 


execution  of  the  fault-tolerance  techniques,  or  of  the  rest  of  the 
hardware  and  software,  or  both.  The  framework  presented  here 
facilitates  control  o*  such  reconfiguration.  It  reduces  system  overhead 
(hard  and  soft)  due  to  fault  tolerance,  and  increases  the  overall  system 
effectiveness. 

Numerous  levels  at  which  these  techniques  for  fault  tolerance  may  be 
applied  are  readily  identifiable.  These  levels  range  from  components  to 
modules  to  processors;  from  bits  of  memory  to  words  to  blocks  to  memory 
modules  to  hierarchies  of  diverse  types  of  memories  (e.g,,  as  in  a 
virtual  memory,  in  which  all  memories  in  the  hierarchy  appear  to  the 
external  interfaces  as  a  single  level);  from  hardware  to  microprogram 
through  various  levels  of  operating  system  software  to  command  software 
"O  user  programs;  from  system  elements  to  systems  to  networks  of 
systems. 

A  very  significant  system  structuve  is  given  by  Dijkstra  /3/.  Details 
internal  to  implementation  at  a  given  level  are  made  invisible  to  all 
higher  levels  by  the  interface  language  at  the  given  level.  Capability 
at  that  level  is  dependent  on  the  capability  of  the  next  lower  level, 
and  is  precisely  that  provided  by  the  interface  language.  The  use  of 
these  distinct  levels  of  invisibility",  or  "levels  of  abstraction",  is 
highly  beneficial  to  system  development.  Two  familiar  examples  are  the 
invisibilility  of  a  cache  memory  to  a  program  and  the  invisibility  of 
multiprogramming  to  a  user. 

THE  HIERARCHICAL  FRAMEWORK 

The  desired  hierarchical  framework  for  fault-  tolerant  computing  systems 
is  as  follows. 

(1)  Various  levels  of  structure  are  established  explicitly  in  the 
design  as  levels  of  invisibility.  Control  and  comunication  facilities 
must  be  provided.  (Useful  mechanisms  are  known  for  this  purpose,  e.g., 
for  coordinating  among  processes  —  both  to  avoid  conflicts  and  to 
permit  sharing  of  programs,  data  and  control  — —  and  for  communicating 
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•song  or  within  levels.  Except  for  deadlock  avoidance,  these  mechanisms 
are  fairly  clear-cut.) 

(2)  Associated  with  these  levels  are  possible  configurations  of 
fault-tolerance  techniques  and  possible  modes  of  dynamic 
reconfiguration . 

(3)  Analysis,  simulation,  and  operating  experience  should  be  used  to 
study  the  relative  effectiveness  of  these  techniques  under  varying 
demands  and  of  reliable  algorithms  for  deciding  how  and  when  to  switch 
among  configurations.  The  suitability  of  the  choice  of  levels  should 
also  be  evaluated. 

An  illustration  of  this  framework  is  provided  by  Table  I.  The  firat 
column  of  the  table  identifies  some  typical  levels  of  invisibility. 

(Lower  levels  are  toward  the  top  of  the  table.)  The  second  column  gives 
examples  of  concepts  invisible  at  each  level.  The  third  column  showa 
techniques  which  can  enhance  fault  tolerance  at  each  level,  and  whose 
details  should  be  invisible  at  higher  levels.  Those  techniques  in  the 
table  which  lend  themselves  to  dynamic  reconfigurability  are  indicated 
by  an  aaterlsk.  The  dynamic  control  over  reconfiguration  of  such 
techniques  may  be  done  Internally,  or  via  the  interface  language  for  the 
appropriate  level.  Techniques  at  one  level  may  be  applied  relatively 
Independently  of  those  at  other  levels,  if  desired. 

As  an  example,  consider  a  system  normally  configured  as  five  independent 
multlprocessed  processors.  At  the  VIRTUAL  SYSTEM  level,  each  user  (or 
application  environment)  deals  with  a  command  language  interface  to  the 
system.  At  the  VIRTUAL  PROCESS  level,  each  virtual  process  may  in  turn 
employ  one  or  more  processes,  either  to  exploit  intrinsic  parallelism 
(e.g.,  simultaneously  processing,  moving  a  file,  and  printing)  or  to 
provide  redundant  computations.  At  the  VIRTUAL  PROCESSOR  level,  these 
processes  may  be  executed  on  the  same  or  on  different  processors.  The 
configuration  might  on  occasion  include  two  processors  in  a  comparison 
mode  with  two  identical  processes  (or  with  two  different  algorithms),  or 
three  processors  in  a  voting  mode,  or  even  in  rare  cases  five  in  a 
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INVISIBILITY  LEVEL 
(examples) 

Components,  chip* 
Module* 

Functlonel  units: 
Processor* , 
central,  etc. 

Memories 

Input-output 
Vlrtuel  herdvare 
Virtual  processor 

Virtual  process 

Vlrtuel  memory 

Virtual  Input- 
output 

Virtual  system 

Virtual  network 


INVISIBLE  CONCEPTS  APPLICABLE  FAULT-TOLERANCE  TECHNIQUES 

(examples)  (examples) 


detail*,  Intrinsically  reliable  technologies,  Rood  engineering, 

fabrication  method*  quality  control,  ceding  end  fault-masking,  replication 

Board  layouts,  pin  Conservative  design,  reliable  connectors,  environmental 

connections,  titling  control;  ‘Diagnosis,  component  replication,  replecamant 

Processor  algorithm*  ‘Automatic  Instruction  retry,  arithmetic  coding 
Address  calculation  Bounds  checking,  memory  protection 

Bua  control  ‘Alternate  routes,  coding,  degradable  priority  mechanisms 

Interrupts  ‘Rece-free  fell-operetlonal  Interrupt  design 


Cecha  mechanisms 
Internal  representation 
Internal  configurations 
Device  characteristic!. 

Medle  properties, 
device  dependence 

Configurations 


Multiprocessing — 
processor  multiplexing 
for  distinct  processes 
Array  computing 
Processor  dispatching 

Multiprog raining — 
process  multiplexing  on 
e  virtual  processor 
Process  scheduling 
Virtual  Interrupts, 
process  Isolation 


Multiplexing  of  memory 
hierarchy:  locations, 
relocations,  backup  end 
retrieval,  directories 


1-0  multiplexing, 
vlrtuel  devlcas 
Exception  handling 
Asynchrony,  buffering 


‘Automatic  reloading 
‘Coding  on  memory  contents 

‘Reconfiguration  around  bed  memory  (via  peglng,  de-lnterlace) 
Use  of  read-only  memories  to  evold  overwrite  and  nld  recovery 

‘Coding  on  contents  of  medle  and  transmission 
‘Verlflcetlon,  checkin?,  rereed  and  compere  after  write 

‘Configuration  sensing  end  s» : f-reconflguretlon,  powering  on- 
off  (e.g.,  spares),  distributing  end  repleclng  power  supplies 

Coding,,  handshaking  on  Interprocessor  romunlcetlon,  evoldance 
of  Interprocessor  Interference;  ‘Replication  of  physical  pro¬ 
cessors  as  e  single  (virtual)  processor,  voting  es  needed 
•Reconfiguration  and  replacement  within  the  arrey 
•Configuration  Insensitivity  vie  checked  teble-drlvlug 

•Replication  of  virtual  processors  for  e  single  process 
‘Independent  computational  checks  (via  possibly  distinct 
procassea)  es  single  virtual  process;  ‘Automatic  rollback 
•Explicit  measures  of  permitted  degradation  per  process 
Sefeguerds  on  Interprocess  connunlcatlon  (vs.  lost  Interrupts, 
blocked  polling),  evoldence  of  Interprocess  Interference, 
lntreprocess  protection  (rings,  capabilities,  master  modes) 

•Replication  of  crltlcel  date  In  verlous  pieces  In  hierarchy, 
Including  reliable  cheap  beckup  store;  ‘Automatic  rollback 
Redundant  pointers  In  directory  structure  and  file  meps  to 
permit  fast  recovery;  Access  control  on  files  (e.g.,  write 
protection)  and  the  uaa  of  pure  procedure  to  Inhibit  loss  of 
crltlcel  deta  or  programs  end  to  aid  In  automatic  rollbeck 

Hendsheklng  to  evold  loss  of  Information;  ‘Status  Information 
•Devlca  swltcheblllty,  media  replication 

‘Coding  (e.g.,  redundant  heedars);  ‘Flexible  error  handling 
Race-condition  end  deedlock  evoldence 


User  multiplexing, 
sharing  of  data 
System  correctness 


System  multiplexing 


Isolation  of  usars  from  the  system  snd  each  other; 
•Controlled  sharing  (If  eny) ;  Self-Identifying  descriptors 
‘Validation,  evaluation  of  effectiveness  end  correctness 
‘On-line  malntenence;  Good  compilers,  diagnostics,  debuggers 

‘Coding  on  Intersyatem  communication,  alternate  petha 
‘Detailed  status  of  network  control  end  network  requests 


TABLE  1 


Examples  of  techniques  for  feult-tolerence  eppllcable  to  verlous  levels  of  Invisibility. 
Asterisks  denote  techniques  particularly  ameneble  to  dynamic  reconfigurability. 


voting  mode.  In  these  modes  there  may  be  4,  3,  and  1  distinct  virtual 
process (es),  respectively,  instead  of  5  as  in  the  fully  multiprocessing 
mode.  (Several  of  these’ modes  are  also  useful  if  some  processors  are 
not  operational,  in  which  case  replacement  is  also  desirable.)  The 
internal  mechanics  of  such  mechanisms  should  be  mostly  invisible  to  each 
virtual  process. 

At  the  VIRTUAL  MEMORY  level,  device  addresses  are  invisible.  When  being 
actively  used,  a  virtual  memory  page  may  in  fact  be  found  in  various 
states  of  recency  and/or  in  various  modes  of  replication  on  various 
devices  in  the  memory  hierarchy,  even  in  the  absence  of  fault-tolerance 
techniques.  For  example,  in  a  paged  environment,  various  Instances  of  a 
given  page  may  exist  simultaneously  in  a  cache- type  memory,  in  primary 
memory  and  in  secondary  memory.  If  it  is  procedure  that  is  "pure" 
(unchanged  by  execution) ,  then  the  contents  of  all  Instances  are 
identical  (barring  errors);  if  it  is  data,  the  Instances  may  differ. 

In  the  present  framework  this  natural  temporary  proliferation  can  be 
used  constructively  to  provide  checkpoints,  thus  greatly  facilitating 
automatic  rollback.  This  is  especially  useful  with  various  Instances  of 
critical  data. 

At  the  MEMORY  level,  coding  techniques  offer  very  inexpensive 
fault-tolerance.  Only  8  redundant  bits  are  needed  to  provide 
single-error  correction  for  64-bit  words  in  memory  and  arithmetic,  a  13% 
Increase  in  memory  cost.  (The  cost  of  error  correcting  circuitry  is 
small  by  comparison.)  Coding  techniques  also  lend  themselves  to  dynamic 
reconfiguration.  One  such  approach  involves  different  uses  of  a 
particular  encoding.  Consider  for  example  a  code  with  Hamming  (or 
arithmetic)  distance  4  for  single-error  correction  and  double-error 
detection.  When  the  multiple  error  rate  is  high,  the  code  may  better  be 
used  for  triple-error  detection  (accompanied  by  increasingly  loud  cries 
for  help).  (Another  example  is  using  a  byte-error  correcting  code  as  a 
multiple-error  detecting  code.)  A  second  approach  involves  varying  the 
encoding  itself,  e.g.,  changing  the  redundancy. 

Similarly  at  the  MODULE  level,  multiple  arithmetic  or  functional  units 
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tied  to  a  control  unit  may  be  used  in  replication  for  fault  tolerance, 
in  synchronsm  as  in  the  ILL1AC  IV  for  handling  parallelism  in 
computation,  or  independently.  The  first  of  these  applications 
substantially  increases  reliability,  while  the  others  may  substantially 
increase  the  computational  throughput. 

Explicit  levels  of  structure  are  now  evident  in  a  few  recent  operating 
systems.  For  example,  the  Multics  protection  hierarchy  (see  /4 /) 
provides  successive  levels  of  resilience  to  errors  in  its  levels  of 
protectability.  A  spectrum  of  criticality  exists  with  respect  to 
faults.  Only  malfunctions  (hard  or  soft)  involving  the  lowest  software 
level  affect  the  viability  of  the  system.  Others  have  diminishingly 
serious  affects  on  the  correctness  of  operation  as  the  level  increases, 
e.g.  aborting  a  user's  process  or  one  command.  As  with  hardware, 
software  techniques  for  fault  tolerance  may  differ  from  level  to  level. 

IMPLICATIONS  OF  THE  HIERARCHICAL  FRAMEWORK 


There  are  numerous  advantages  of  this  hierarchical  framework.  These 
include  considerations  of  reliability,  computational  capacity,  and  cost. 

O)  A  wide  variety  of  techniques  can  be  applied,  each  where  it  is  most 
effective,  responsive  to  the  needs  for  fault  tolerance  and  computing 
capacity,  and  subject  to  the  cost  factors.  Each  configuration  can  be 
dynamically  altered,  based  on  the  current  usage  of  the  system.  (This 
may  affect  more  than  one  level  at  once.)  The  net  cost  of  system  fault 
tolerance  can  therefore  be  reduced,  especially  if  rarely  used  techniques 
can  be  performed  reliably  in  software.  Considerable  savings  also  result 
if  occasional  modest  real-time  delays  are  permitted  U.g.,  for 
diagnosis,  recovery  and  reconfiguration),  further  reducing  the  need  for 
dedicated  hardware.  Nonuniform  costs  also  permit  the  reduction  of  the 
incremental  cost  of  fault  tolerance.  If  memory  costs  (including 
secondary  storage)  dominate  total  hardware  costs,  then  the  relatively 
small  cost  of  redundancy  in  memory  (e.g.,  logarithmic  for  single-error 
correction  in  memory  and  arithmetic)  may  dominate  the  incremental  cost, 
even  with  replicated  processors.  If  memory  costs  do  not  dominate,  then 
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memory  is  relatively  cheap  and  logic-in-raemory  architectures  (see  below) 
may  be  of  interest.  The  framework  also  facilitates  checkpoint 
mechanisms  which  permit  varying  degrees  of  rollback  as  needed,  involving 
different  levels  of  the  hierarchy.  On-site  maintenance  is  also  aided, 
as  are  on-line  interactive  diagnostics. 

(2)  In  general,  computing  capacity  not  currently  dedicated  to  fault 
tolerance  is  available  for  useful  computing,  assuming  reasonable  system 
balance.  It  is  desirable  to  have  pools  of  modules,  of  functional  units, 
of  processors,  and  of  systems  to  configure  among.  The  multiplicity  of 
each  pool  should  be  large  enough  so  that  the  mesh  of  graceful 
degradation  is  reasonably  smooth  and  that  the  loss  of  any  unit  is  not 
serious.  This  increases  the  overall  system  effectiveness,  in  terms  of 
both  computing  capacity  and  fault  tolerance. 

(3)  The  intrinsic  structure  of  the  hierarchy  enhances  each  stage  of 
system  development,  including  the  stages  of  designing,  implementing, 
documenting,  debugging,  certifying,  analyzing,  maintaining,  and 
modifying  a  system.  At  each  such  stage  the  notion  of  levels  of 
invisibility  permits  issues  of  fault  tolerance  relevant  to  lower  levels 
to  be  abstracted  and  analyzed,  aiding  in  isolating  any  side-effects. 

Thus  the  framework  serves  as  a  useful  model  as  well. 

(4)  Recent  technological  advantages  (e.g.,  LSI)  significantly  improve 
the  cost-effectiveness  of  many  of  the  techniques.  These  advances  should 
also  stimulate  new  architectural  directions,  such  as  multiprocessors 
with  considerable  multiplicity,  and  distributed-logic  or  logic-in-memory 
designs.  The  latter  case  involves  large  arrays  of  small  memory 
elements,  each  containing  processing  capability.  These  arrays  may  be 
organized  into  subarrays  of  subarrays,  possibly  with  structures 
geometrically  oriented  toward  the  problem  to  be  solved. 

There  are  of  course  many  questions  left  unanswered, 

(1)  Questions  of  overhead  and  reliability  resulting  from  the  control  of 
such  systems  must  be  examined  carefully.  It  appears  that  the  overhead 
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can  usually  be  kept  small,  except  when  fault- tolerance  limits  are 
approached.  It  is  obviously  desirable  that  the  mechanisms  for 
controlling  reconfiguration  must  themselves  be  fault  tolerant,  thrash 
resistant,  and  reconfigurable.  Interference  problems  and 
intercommunication  must  also  be  handled  reliably. 

(2)  This  framework  seems  particularly  effective  for  large 
general-purpose  systems.  How  effective  it  can  be  under  various 
circumstances,  e.g.,  for  small  systems,  for  those  with  tight  real-time 
constraints,  requires  further  study. 

(3)  How  can  the  various  tradeoffs  among  fault  tolerance,  computing 
capacity,  cost,  overhead,  etc.,  be  chatacterized?  Under  what 
circumstances  is  it  desirable  to  reconfigure?  What  kind  of  limiting 
behavior  occurs  as  computing  capacity  or  fault-tolerance  capacity  is 
reached?  What  are  the  penalties  associated  with  having  too  many  or  too 
few  levels?  What  happens  to  the  notion  of  the  "weakest  link"?  Can  it 
be  distributed  among  less  weak  links?  How  does  it  shift  during 
reconfiguration? 

THE  MASSIVE  TRANSIENT  RECOVERY  PROBLEM 


As  an  example  of  a  specific  problem  which  can  he  greatly  simplified  by 
the  adoption  of  the  hierarchical  framework,  consider  the  "massive- 
transient"  recovery  problem: 

A  correlated  fault  source  (e.g., a  power  surge  or  a  bolt  of  lightning) 
has  left  all  units  of  the  system  suspect,  perhaps  introducing  both 
transient  and  permanent  faults.  The  problem  is  for  the  system  to 
diagnose  and  configure  itself  back  into  a  working  configuration  and  to 
validate  itself  for  correctness,  all  under  Its  own  control. 

This  problem  is  essentially  a  generalized  fault-  tolerance  problem, 
where  performance  may  cease  temporarily  during  and  just  after  the 
massive  transient.  It  is  also  closely  related  to  normal  system 
initialization.  The  hierarchical  framework  and  the  dynamic 
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reconfigurability  bo.'h  aid  greatly  in  solving  this  problem.  One 
solution  involves  reestablishing  minimally  correct  hardware  by 
bootstrapping  upwards  from  the  lowest  levels  of  the  hierarchy,  until  a 
satisfactory  rudimentary  system  is  obtained.  It  is  also  desirable  to 
validate  downwards  from  the  higher  levels.  This  solution  is  aided  by 
the  us?j  of  a  hard-wired  non-volatile  read-only  memory  which  provides  a 
basis  of  correct  programs  for  recovery.  Further  help  is  offered  if  this 
memory  is  directly  executable  and  the  programs  are  pure,  and  if  these 
programs  operate  only  cut  of  local  memory  at  first.  By  working  up  the 
hierarchy,  valid  portions  of  the  system  begin  to  emerge.  (Another 
solution  might  involve  trying  experiments  on  various  configurations  of 
the  whole  system.)  Note  that  this  problem  may  be  intrinsically 
insoluble  for  a  given  system.  It  may  also  oe  insoluble  for  the 
particular  massive  transient,  e.g.,  because  not  enough  operational 
equipment  remains  to  self-diagnose  and  configure  a  valid  system,  or  even 
Just  to  operate  such  a  system.  (More  equipment  might  be  required  for 
diagnosis  than  for  operation.) 

CONCLUSIONS 

The  hierarchical  framework  presented  here  appears  to  have  great 
potential  in  the  design  of  fault-tolerant  systems.  It  can  increase  the 
effectiveness  of  new  systems  as  well  as  the  ease  and  flexibility  of 
their  development  and  operation.  It  should  increase  in  utility  as 
technological  advances  permit  much  larger  systems  to  be  developed. 
Further  study  is  intended. 
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